How to use OpenCV 4 on Android and view camera live view

About this article

Create a simple Android application that just inverts the camera image negatively and positively and outputs it. OpenCV4 is used for camera input. It can be used as a base project for creating some kind of application using camera images. In this article, I will explain how to include the OpenCV library in the APK without using OpenCV Manager.

image.png

** Addendum (2020/07/18) ** It seems that this method cannot be used with recent Android Studio (or Android SDK). It is better to use the Camera2 API or Camera X. Here's how to use CameraX. https://qiita.com/iwatake2222/items/c0ebe6d84afdef57aab3

Development environment

Advance preparation

Android Studio installation

https://developer.android.com/studio

It may be convenient to put the path in the folder with adb. (Not used in this article)

Download OpenCV

https://github.com/opencv/opencv/releases Download ʻopencv-4.1.0-android-sdk.zip` from and unzip it to a suitable location. Please choose the version you like.

Hereafter, the explanation will be described with the extraction destination path as {OpenCV-android-sdk}.

Incorporate OpenCV into your project and apk

Create a new project

--Open Android Studio --Click Start a new Android Studio project --Select ʻEmpty Activityand Next --Set application name etc. --If you just want to try it, all defaults are fine --This time, Java was used as the development language, and API 21 was set as the Minimum API level. --If you plan to release it, please decide the package name properly --Connect your Android device to your PC via USB --SelectRun-> Run'App'` from the menu bar, and it is OK if the application in the default state where Hello World is output starts on the Android device.

01.png

Import OpenCV

02.png

--Dependency settings --Select File-> Project Structure from the menu bar --Select ʻapp on the left and select the Dependenciestab --Click+ (add)in the upper right and select3: Module dependenciy --Select ʻopencv and OK

03.png

-(Not required) Copy of OpenCV library (so file) --In the past, I had to copy the OpenCV library (so file) to jniLibs by myself, but now (OpenCV4 or later ??) it seems unnecessary.

-(Not required) Fix AndroidManifest, build.gradle --In the past, it was necessary to delete targetSdkVersion etc. specified in OpenCV's ʻAndroidManifest.xml, or to align various sdkVersions (compileSdkVersionetc.) inbuild.gradle` with app. had. --Now (OpenCV4 or later ??) seems unnecessary

OpenCV loading

Code of Activity that uses OpenCV (If you selected Empty Activity at the time of project generation and did not change any settings, ʻapp \ src \ main \ java \ com \ example \ myapplication \ MainActivity.java`) as follows Edit

In practice, either one is sufficient. In a real product, I would only use System.loadLibrary.

MainActivity.java


public class MainActivity extends AppCompatActivity {

    static {
        System.loadLibrary("opencv_java4");     //add to
    }

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);

        OpenCVLoader.initDebug();	//add to
    }
}

Run in this state, and if Logcat shows the following log, it's OK.

Logcat output


2019-05-25 18:50:20.050 29439-29439/com.example.myapplication I/OpenCV/StaticHelper: General configuration for OpenCV 4.1.0 =====================================
2019-05-25 18:50:20.050 29439-29439/com.example.myapplication I/OpenCV/StaticHelper:   Version control:               4.1.0
2019-05-25 18:50:20.050 29439-29439/com.example.myapplication I/OpenCV/StaticHelper:   Platform:
2019-05-25 18:50:20.050 29439-29439/com.example.myapplication I/OpenCV/StaticHelper:     Timestamp:                   2019-04-07T19:03:43Z
2019-05-25 18:50:20.050 29439-29439/com.example.myapplication I/OpenCV/StaticHelper:     Host:                        Linux 4.15.0-47-generic x86_64
2019-05-25 18:50:20.050 29439-29439/com.example.myapplication I/OpenCV/StaticHelper:     Target:                      Android 1 aarch64
2019-05-25 18:50:20.050 29439-29439/com.example.myapplication I/OpenCV/StaticHelper:     CMake:                       3.6.0-rc2
2019-05-25 18:50:20.050 29439-29439/com.example.myapplication I/OpenCV/StaticHelper:     CMake generator:             Ninja
2019-05-25 18:50:20.050 29439-29439/com.example.myapplication I/OpenCV/StaticHelper:     CMake build tool:            /opt/android/android-sdk.gradle/cmake/3.6.4111459/bin/ninja
2019-05-25 18:50:20.050 29439-29439/com.example.myapplication I/OpenCV/StaticHelper:     Configuration:               Release

Somehow, when I looked at the top of Logcat, ʻOpenCV / StaticHelper: OpenCV error: Cannot load info library for OpenCV` I got this error, but it did not affect the operation.

Camera live view display

Now that you have a project that incorporates OpenCV, let's create an application that displays the live view of the camera.

Permission settings

Make settings to use the camera

Manifest settings

First, you need to declare "This application uses a camera".

Add the following three lines to ʻapp / src / main / AndroidManifest.xml. Add it directly under . Add it in the same row as the existing `.

AndroidManifest.xml


<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
    package="com.example.myapplication">

    <!--↓ Add-->
    <uses-permission android:name="android.permission.CAMERA"/>
    <uses-feature android:name="android.hardware.camera" android:required="false"/>
    <uses-feature android:name="android.hardware.camera.autofocus" android:required="false"/>
    <!--↑ Add-->

    <application
        android:allowBackup="true"
abridgement
    </application>

</manifest>

Get permission

In fact, just setting the manifest is enough. However, this alone requires the user to set permissions from the app's admin screen. Normal users don't even know where the screen is.

Therefore, the app will voluntarily display a request screen saying "Please give me permission".

With the following code, if permission is not set, the permission request screen will be displayed. The calling code will be shown later.

MainActivity.java


    public static boolean getPermissionCamera(Activity activity) {
        if (ContextCompat.checkSelfPermission(
                activity,
                android.Manifest.permission.CAMERA)
                != PackageManager.PERMISSION_GRANTED) {
            String[] permissions = new String[]{Manifest.permission.CAMERA};
            ActivityCompat.requestPermissions(
                    activity,
                    permissions,
                    0);
            return false;
        } else {
            return true;
        }
    }

CameraView placement

Add CameraView to the layout. Open layout / activity_main.xml and in Text mode, add ʻorg.opencv.android.JavaCameraView`.

activity_main.xml


<?xml version="1.0" encoding="utf-8"?>
<android.support.constraint.ConstraintLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    tools:context=".MainActivity">

    <TextView
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:text="Hello World!"
        app:layout_constraintBottom_toBottomOf="parent"
        app:layout_constraintLeft_toLeftOf="parent"
        app:layout_constraintRight_toRightOf="parent"
        app:layout_constraintTop_toTopOf="parent" />

    <!--↓ Add-->
    <org.opencv.android.JavaCameraView
        android:id="@+id/camera_view"
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        android:visibility="visible"
        tools:layout_editor_absoluteX="0dp"
        tools:layout_editor_absoluteY="0dp"
        app:show_fps="true"/>
    <!--↑ Add-->
</android.support.constraint.ConstraintLayout>

Full code

The full code is shown below.

--Inherits the CameraBridgeViewBase.CvCameraViewListener interface. --After adding ʻimplements CameraBridgeViewBase.CvCameraViewListener to MainActivity, press Alt + Enter to add import appropriately. Press Alt + Enter again to do Implement Methods to add the required functions. --Declare CameraBridgeViewBase and assign it to the JavaCameraViewyou placed in the layout earlier. Specify this Activity withsetCvCameraViewListener to call the ʻonCameraViewStarted, ʻonCameraViewStopped, and ʻonCameraFrame added earlier. --ʻOnCameraFrame is a function called every frame. The input image from the camera is entered in the argument (ʻinputFrame), and the Mat to be displayed is set in the return value. ――Since this ʻonCameraFrame runs in a thread separate from the UI thread, it seems that it is okay to do some heavy processing such as image processing here. I don't know if it's Best Practice. ――This time I tried to reverse the negative and positive --If the processing in ʻonCameraFrame becomes heavy, the fps of drawing will naturally slow down. This is nice for implementers. It seems that the ʻonCameraFrame function will not be called again while doing heavy processing inside the ʻonCameraFrame. So it doesn't seem necessary to be reentrant.

MainActivity.java


public class MainActivity extends AppCompatActivity implements CameraBridgeViewBase.CvCameraViewListener {
    private CameraBridgeViewBase m_cameraView;

    static {
        System.loadLibrary("opencv_java4");
    }

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);

        getPermissionCamera(this);
//        OpenCVLoader.initDebug();
        m_cameraView = findViewById(R.id.camera_view);
        m_cameraView.setCvCameraViewListener(this);
        m_cameraView.enableView();
    }

    public static boolean getPermissionCamera(Activity activity) {
        if (ContextCompat.checkSelfPermission(
                activity,
                android.Manifest.permission.CAMERA)
                != PackageManager.PERMISSION_GRANTED) {
            String[] permissions = new String[]{Manifest.permission.CAMERA};
            ActivityCompat.requestPermissions(
                    activity,
                    permissions,
                    0);
            return false;
        } else {
            return true;
        }
    }

    @Override
    public void onCameraViewStarted(int width, int height) {

    }

    @Override
    public void onCameraViewStopped() {

    }

    @Override
    public Mat onCameraFrame(Mat inputFrame) {
        //Do some image processing here
        //Try to reverse the negative and positive
        Core.bitwise_not(inputFrame, inputFrame);
        return inputFrame;
    }
}

The screen may rotate depending on the model, device orientation, and settings. These adjustments complicate the code and description and are omitted. Similarly, the end and return in onDestory and onResume are also omitted.

in conclusion

--Summary of steps to use OpenCV4 on Android --Summary of steps to get and display camera preview using OpenCV4

Recommended Posts

How to use OpenCV 4 on Android and view camera live view
How to "hollow" View on Android
How to use EventBus3 and ThreadMode
How to use Ruby on Rails
How to use equality and equality (how to use equals)
How to use Bio-Formats on Ubuntu 20.04
How to install and use Composer on an ECS instance on Ubuntu 16.04
A memorandum on how to use Eclipse
How to use binding.pry for view files
How to use Apache Derby on Eclipse
How to use OrientJS and OrientDB together
[Ruby on Rails] How to use CarrierWave
How to detect microphone conflicts on Android
How to set up and use kapt
[Ruby on Rails] How to use redirect_to
[Ruby on Rails] How to use kaminari
How to use substring and substr methods
How to use @Builder and @NoArgsConstructor together
How to use ExpandableListView in Android Studio
[Android] How to turn the Notification panel on and off using StatusBarManager
How to use the camera module OV7725 (ESP32-WROVER-B)
[Java] How to use FileReader class and BufferedReader class
[Ruby] How to use gsub method and sub method
Notes on how to use each JUnit Rule
How to use Segmented Control and points to note
How to use scope and pass processing (Jakarta)
[Ruby on Rails] How to use session method
[Java] How to use Calendar class and Date class
How to use Eclipse on my PC with 32bit and 2GB of memory
How to use Map
How to use rbenv
How to use letter_opener_web
How to use with_option
How to use fields_for
How to use java.util.logging
How to use map
How to use collection_select
How to use Twitter4J
How to use active_hash! !!
How to use MapStruct
How to use hidden_field_tag
How to use TreeSet
[How to use label]
How to use identity
How to use hashes
How to use JUnit 5
How to use Dozer.mapper
How to use Gradle
How to use org.immutables
How to use java.util.stream.Collector
How to use VisualVM
How to use Map
How to use java non-standard library on IntelliJ IDEA
How to use Truth (assertion library for Java / Android)
How to hover and click on Selenium DOM elements
[Android Studio] I want to use Maven library on Android
How to use RealSense with ubuntu 20.04 and ROS Noetic
[Rails] How to edit and customize devise view and controller
(Ruby on Rails6) How to create models and tables
How to use and apply Java's JFrame / Canvas class
Notes on how to use regular expressions in Java