How to use OpenCV 4 on Android and view camera live view

About this article

Create a simple Android application that just inverts the camera image negatively and positively and outputs it. OpenCV4 is used for camera input. It can be used as a base project for creating some kind of application using camera images. In this article, I will explain how to include the OpenCV library in the APK without using OpenCV Manager.

** Addendum (2020/07/18) ** It seems that this method cannot be used with recent Android Studio (or Android SDK). It is better to use the Camera2 API or Camera X. Here's how to use CameraX. https://qiita.com/iwatake2222/items/c0ebe6d84afdef57aab3

Development environment

• Windows 10 64-bit
• Android Studio 3.3.2 --Development language: Java
• OpenCV 4.1.0 --Target device: Galaxy S7

Advance preparation

Android Studio installation

https://developer.android.com/studio

It may be convenient to put the path in the folder with adb. (Not used in this article)

Download OpenCV

https://github.com/opencv/opencv/releases Download ʻopencv-4.1.0-android-sdk.zip` from and unzip it to a suitable location. Please choose the version you like.

Hereafter, the explanation will be described with the extraction destination path as {OpenCV-android-sdk}.

Incorporate OpenCV into your project and apk

Create a new project

--Open Android Studio --Click Start a new Android Studio project --Select ʻEmpty Activityand Next --Set application name etc. --If you just want to try it, all defaults are fine --This time, Java was used as the development language, and API 21 was set as the Minimum API level. --If you plan to release it, please decide the package name properly --Connect your Android device to your PC via USB --SelectRun-> Run'App'` from the menu bar, and it is OK if the application in the default state where Hello World is output starts on the Android device.

Import OpenCV

• Import Module --Select File-> New-> ʻImport Module from the menu bar --Specify {OpenCV-android-sdk} / sdk, which was downloaded and expanded earlier, in Source directory. --Specify {OpenCV-android-sdk} / sdk / java in OpenCV 3.x or earlier? --If Module name is the default, it will be sdk and it will be difficult to understand, so set it appropriately as ʻopencv. --Other settings are left as default, Next, Finish`

--Dependency settings --Select File-> Project Structure from the menu bar --Select ʻapp on the left and select the Dependenciestab --Click+ (add)in the upper right and select3: Module dependenciy --Select ʻopencv and OK

-(Not required) Copy of OpenCV library (so file) --In the past, I had to copy the OpenCV library (so file) to jniLibs by myself, but now (OpenCV4 or later ??) it seems unnecessary.

-(Not required) Fix AndroidManifest, build.gradle --In the past, it was necessary to delete targetSdkVersion etc. specified in OpenCV's ʻAndroidManifest.xml, or to align various sdkVersions (compileSdkVersionetc.) inbuild.gradle` with app. had. --Now (OpenCV4 or later ??) seems unnecessary

OpenCV loading

Code of Activity that uses OpenCV (If you selected Empty Activity at the time of project generation and did not change any settings, ʻapp \ src \ main \ java \ com \ example \ myapplication \ MainActivity.java`) as follows Edit

In practice, either one is sufficient. In a real product, I would only use System.loadLibrary.

MainActivity.java

public class MainActivity extends AppCompatActivity {

    static {
        System.loadLibrary("opencv_java4");     //add to
    }

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);

        OpenCVLoader.initDebug();	//add to
    }
}

Run in this state, and if Logcat shows the following log, it's OK.

Logcat output

2019-05-25 18:50:20.050 29439-29439/com.example.myapplication I/OpenCV/StaticHelper: General configuration for OpenCV 4.1.0 =====================================
2019-05-25 18:50:20.050 29439-29439/com.example.myapplication I/OpenCV/StaticHelper:   Version control:               4.1.0
2019-05-25 18:50:20.050 29439-29439/com.example.myapplication I/OpenCV/StaticHelper:   Platform:
2019-05-25 18:50:20.050 29439-29439/com.example.myapplication I/OpenCV/StaticHelper:     Timestamp:                   2019-04-07T19:03:43Z
2019-05-25 18:50:20.050 29439-29439/com.example.myapplication I/OpenCV/StaticHelper:     Host:                        Linux 4.15.0-47-generic x86_64
2019-05-25 18:50:20.050 29439-29439/com.example.myapplication I/OpenCV/StaticHelper:     Target:                      Android 1 aarch64
2019-05-25 18:50:20.050 29439-29439/com.example.myapplication I/OpenCV/StaticHelper:     CMake:                       3.6.0-rc2
2019-05-25 18:50:20.050 29439-29439/com.example.myapplication I/OpenCV/StaticHelper:     CMake generator:             Ninja
2019-05-25 18:50:20.050 29439-29439/com.example.myapplication I/OpenCV/StaticHelper:     CMake build tool:            /opt/android/android-sdk.gradle/cmake/3.6.4111459/bin/ninja
2019-05-25 18:50:20.050 29439-29439/com.example.myapplication I/OpenCV/StaticHelper:     Configuration:               Release

Somehow, when I looked at the top of Logcat, ʻOpenCV / StaticHelper: OpenCV error: Cannot load info library for OpenCV` I got this error, but it did not affect the operation.

Camera live view display

Now that you have a project that incorporates OpenCV, let's create an application that displays the live view of the camera.

Permission settings

Make settings to use the camera

Manifest settings

First, you need to declare "This application uses a camera".

Add the following three lines to ʻapp / src / main / AndroidManifest.xml. Add it directly under . Add it in the same row as the existing `.

AndroidManifest.xml

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
    package="com.example.myapplication">

    <!--↓ Add-->
    <uses-permission android:name="android.permission.CAMERA"/>
    <uses-feature android:name="android.hardware.camera" android:required="false"/>
    <uses-feature android:name="android.hardware.camera.autofocus" android:required="false"/>
    <!--↑ Add-->

    <application
        android:allowBackup="true"
abridgement
    </application>

</manifest>

Get permission

In fact, just setting the manifest is enough. However, this alone requires the user to set permissions from the app's admin screen. Normal users don't even know where the screen is.

Therefore, the app will voluntarily display a request screen saying "Please give me permission".

With the following code, if permission is not set, the permission request screen will be displayed. The calling code will be shown later.

MainActivity.java

    public static boolean getPermissionCamera(Activity activity) {
        if (ContextCompat.checkSelfPermission(
                activity,
                android.Manifest.permission.CAMERA)
                != PackageManager.PERMISSION_GRANTED) {
            String[] permissions = new String[]{Manifest.permission.CAMERA};
            ActivityCompat.requestPermissions(
                    activity,
                    permissions,
                    0);
            return false;
        } else {
            return true;
        }
    }

CameraView placement

Add CameraView to the layout. Open layout / activity_main.xml and in Text mode, add ʻorg.opencv.android.JavaCameraView`.

activity_main.xml

<?xml version="1.0" encoding="utf-8"?>
<android.support.constraint.ConstraintLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    tools:context=".MainActivity">

    <TextView
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:text="Hello World!"
        app:layout_constraintBottom_toBottomOf="parent"
        app:layout_constraintLeft_toLeftOf="parent"
        app:layout_constraintRight_toRightOf="parent"
        app:layout_constraintTop_toTopOf="parent" />

    <!--↓ Add-->
    <org.opencv.android.JavaCameraView
        android:id="@+id/camera_view"
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        android:visibility="visible"
        tools:layout_editor_absoluteX="0dp"
        tools:layout_editor_absoluteY="0dp"
        app:show_fps="true"/>
    <!--↑ Add-->
</android.support.constraint.ConstraintLayout>

Full code

The full code is shown below.

--Inherits the CameraBridgeViewBase.CvCameraViewListener interface. --After adding ʻimplements CameraBridgeViewBase.CvCameraViewListener to MainActivity, press Alt + Enter to add import appropriately. Press Alt + Enter again to do Implement Methods to add the required functions. --Declare CameraBridgeViewBase and assign it to the JavaCameraViewyou placed in the layout earlier. Specify this Activity withsetCvCameraViewListener to call the ʻonCameraViewStarted, ʻonCameraViewStopped, and ʻonCameraFrame added earlier. --ʻOnCameraFrame is a function called every frame. The input image from the camera is entered in the argument (ʻinputFrame), and the Mat to be displayed is set in the return value. ――Since this ʻonCameraFrame runs in a thread separate from the UI thread, it seems that it is okay to do some heavy processing such as image processing here. I don't know if it's Best Practice. ――This time I tried to reverse the negative and positive --If the processing in ʻonCameraFrame becomes heavy, the fps of drawing will naturally slow down. This is nice for implementers. It seems that the ʻonCameraFrame function will not be called again while doing heavy processing inside the ʻonCameraFrame. So it doesn't seem necessary to be reentrant.

MainActivity.java

public class MainActivity extends AppCompatActivity implements CameraBridgeViewBase.CvCameraViewListener {
    private CameraBridgeViewBase m_cameraView;

    static {
        System.loadLibrary("opencv_java4");
    }

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);

        getPermissionCamera(this);
//        OpenCVLoader.initDebug();
        m_cameraView = findViewById(R.id.camera_view);
        m_cameraView.setCvCameraViewListener(this);
        m_cameraView.enableView();
    }

    public static boolean getPermissionCamera(Activity activity) {
        if (ContextCompat.checkSelfPermission(
                activity,
                android.Manifest.permission.CAMERA)
                != PackageManager.PERMISSION_GRANTED) {
            String[] permissions = new String[]{Manifest.permission.CAMERA};
            ActivityCompat.requestPermissions(
                    activity,
                    permissions,
                    0);
            return false;
        } else {
            return true;
        }
    }

    @Override
    public void onCameraViewStarted(int width, int height) {

    }

    @Override
    public void onCameraViewStopped() {

    }

    @Override
    public Mat onCameraFrame(Mat inputFrame) {
        //Do some image processing here
        //Try to reverse the negative and positive
        Core.bitwise_not(inputFrame, inputFrame);
        return inputFrame;
    }
}

The screen may rotate depending on the model, device orientation, and settings. These adjustments complicate the code and description and are omitted. Similarly, the end and return in onDestory and onResume are also omitted.

in conclusion

--Summary of steps to use OpenCV4 on Android --Summary of steps to get and display camera preview using OpenCV4