Run Tensorflow with THETA V

Introduction

Tensorflow has a sample that runs on Android, and it is also introduced on the following sites. https://qiita.com/icchi_h/items/a1df9f27569714edfc5e Here, Tensorflow is operated with the spherical camera THETA V equipped with Android.

(2018/8/18: Added the article that was divided into the front and back parts. Here Go to 2018/10/20 There is a commentary material (PPT) at the event that was held.)

Development environment

procedure

Source code acquisition (Tensorflow, Plugin SDK)

Get tensorflow from github and prepare the source code. here, A forked version of https://github.com/tensorflow/tensorflow (commit: 26353f9b51091312e7097143aee9c2d05e2011fd) was placed at https://github.com/mktshhr/tensorflow-theta and updated for THETA V. The tag "Qiita20180818" is the source code corresponding to this article. Among them, https://github.com/mktshhr/tensorflow-theta/tree/Qiita20180818/tensorflow/examples/android is the scope of change. After getting the source code, you can try it by opening / tensorflow-theta / examples / android in Android Studio. The changes described below have already been applied.

git clone https://github.com/mktshhr/tensorflow-theta.git 
cd ./tensorflow-theta/examples/android

Open in Android Studio

Open the android folder in Android Studio. Follow the screen by clicking "Add Google Maven repository and sync project". Rewrite line 45 of build.gradle from "bazel" to "none".

// set to 'bazel', 'cmake', 'makefile', 'none'
def nativeBuildSystem = 'none'

Similarly, rewrite line 194 of build.gradle from "compile" to "implementation".

dependencies {
    if (nativeBuildSystem == 'cmake' || nativeBuildSystem == 'none') {
        implementation 'org.tensorflow:tensorflow-android:+'
    }
}

Update with "Sync Now".

Run on Android smartphone

With the changes so far, it works on Android smartphones (Moto G5 Plus, Android 7.0). You can select "Run"-"Debug'TensorflowAndroid'" to build and run it on your Android smartphone. (In the case of THETA V, the camera shuts down with a beeping beeping.)

TensorFlow Demo consists of four activities (apps): TF Classify, TF Detect, TF Stylize, and TF Speech.

Changes for THETA V

Fixed camera common parts (TF Classify, TF Detect, TF Stylize)

Changed to broadcast "com.theta360.plugin.ACTION_MAIN_CAMERA_CLOSE" in onCreate of CameraActivity class. Free up THETA's camera resources and make them available in the Tensorflow app.

  @Override
  protected void onCreate(final Bundle savedInstanceState) {
    LOGGER.d("onCreate " + this);
    sendBroadcast(new Intent("com.theta360.plugin.ACTION_MAIN_CAMERA_CLOSE"));
    super.onCreate(null);

In order to build, add the following to the beginning of the CameraActivity.java file.

import android.content.Intent;

Corrected the onPreviewSizeChosen argument near line 124. I did not rotate.

onPreviewSizeChosen(new Size(previewSize.width, previewSize.height), 0);

Changed around line 99 as follows. Here, RicMoviePreview1024 is set, but settings such as RicMoviePreview3840 are also possible.

            //camera.setDisplayOrientation(90);
            parameters.set("RIC_SHOOTING_MODE", "RicMoviePreview1024");

Changed line 109 as follows. Swap the width and height.

          camera.addCallbackBuffer(new byte[ImageUtils.getYUVByteSize(s.width, s.height)]);
          textureView.setAspectRatio(s.width, s.height);

With the above changes, we were able to confirm the operation of TF Classify and TF Detect on Vysor. I fixed it because the display became a portrait image, but while the new Camera 2 API was available on the Android smartphone (Moto G5 Plus), the operation was different between the smartphone and THETA because THETA uses the legacy Camera API. It may be.

By default, TF Detect uses TF_OD_API (Tensorflow Object Detection API) for object detection.

TF Stylize fix

After making the image a 1: 1 square image and applying Style, modify it so that it returns to a 2: 1 equirectangular image.

Set not to save the aspect ratio when creating a frameToCropTransform transformation matrix with processImage () near line 495 of StylizeActivity.java. Now you have a 2: 1-> 1: 1 image for Style conversion.

  @Override
  protected void processImage() {
    if (desiredSize != initializedSize) {
      LOGGER.i(
          "Initializing at size preview size %dx%d, stylize size %d",
          previewWidth, previewHeight, desiredSize);

      rgbFrameBitmap = Bitmap.createBitmap(previewWidth, previewHeight, Config.ARGB_8888);
      croppedBitmap = Bitmap.createBitmap(desiredSize, desiredSize, Config.ARGB_8888);
      frameToCropTransform = ImageUtils.getTransformationMatrix(
          previewWidth, previewHeight,
          desiredSize, desiredSize,
          sensorOrientation, false);

In Runnable () near line 520 of StylizeActivity.java, resize using Canvas and change the styled croppedBitmap image with stylizeImage () from 1: 1 to 2: 1. You can create a 2: 1 textureCopyBitmap image with a modified style.

    runInBackground(
        new Runnable() {
          @Override
          public void run() {
            cropCopyBitmap = Bitmap.createBitmap(croppedBitmap);
            final long startTime = SystemClock.uptimeMillis();
            stylizeImage(croppedBitmap);
            lastProcessingTimeMs = SystemClock.uptimeMillis() - startTime;
            textureCopyBitmap = Bitmap.createBitmap(previewWidth, previewHeight, Config.ARGB_8888);

            final Paint paint = new Paint();
            paint.setFilterBitmap(true);
            final Canvas canvas = new Canvas(textureCopyBitmap);
            canvas.drawBitmap(croppedBitmap, cropToFrameTransform, paint);

            if (SAVE_PREVIEW_BITMAP) {
              ImageUtils.saveBitmap(textureCopyBitmap, "stylizeImage.png ");
            }
            requestRender();
            readyForNextImage();
          }
        });

At this point, TF Stylize will work.

TF Speech fix

  @Override
  protected void onCreate(Bundle savedInstanceState) {
    // Set up the UI.
    super.onCreate(savedInstanceState);
    
    AudioManager am = (AudioManager) getSystemService(Context.AUDIO_SERVICE); // for THETA
    am.setParameters("RicUseBFormat=false"); // for THETA
    
    setContentView(R.layout.activity_speech);

Added android.permission.MODIFY_AUDIO_SETTINGS permission in AndroidManifest.xml (below is near line 26). The default sampling rate for TF Speech was 16kHz, and the default for THETA was 44.1kHz. It seems that the TF Speech filter does not work well at 44.1kHz, so I operated it at 16kHz, but it seems that it was necessary to add permissions to operate with this setting.

    <uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS" />

With the above changes, it is now possible to recognize voice as much as a smartphone.

Summary

An Android sample of Tensorflow was run on THETA to perform image recognition and voice recognition. TF Classify is an object classifier, but it is easy to misidentify an object because it has a super wide angle in spherical images. TF Detect is an object detector that searches for objects in an image. Since TF Stylize could not apply the style to the 2: 1 image well, I avoided it by applying the style to the 1: 1 image once. TF Speech could be operated by adding the MODIFY_AUDIO_SETTINGS privilege. It is thought that there is still much room for ingenuity, such as improving recognition accuracy for spherical images. The key TensorFlowInferenceInterface class uses trained models (Protocol Buffers (.pb) format files) under assets as parameters. You can try various learning models by arranging and reading in the same way.

reference

https://codelabs.developers.google.com/codelabs/tensorflow-style-transfer-android/index.html https://api.ricoh/docs/theta-plugin-reference/camera-api/ http://iti.hatenablog.jp/entry/2017/05/25/093328

Recommended Posts

Run Tensorflow with THETA V
Run Pico with docker
Run Payara with Docker
Run batch with docker-compose with Java batch
Run TAO Core with Docker
Run LIFF with Spring Boot
Run Rails whenever with docker
Run Java VM with WebAssembly
Run lambda with custom docker image
Run SQL Server with Docker ToolBox
Run nginx + express with fargate 4 core
ubuntu20.04 ・ Run RTAB-MAP with ROS Noetic (1)
Make JupyterLab run anywhere with docker
Run an application made with Java8 with Java6