How to pass image pixel information natively from Unity

There was a case where I wanted to pass the pixel information of a texture image such as Unity's RenderTexture natively, so I had a lot of trouble (especially on the Android side), so make a note. By the way, Android is shit. I want you to disappear quickly. To be more specific, it is a case where you want to pass the image data to the Java library using WebCamera on the Unity side.

Click here for sample project https://bitbucket.org/HoshiyamaTakaaki/pixelreadstest

First of all, implementation on the Unity side

CameraOnPost.cs


RenderTexture.active = m_TargetTexture;
m_TargetTexture2D.ReadPixels(new Rect(0, 0, m_TargetTexture.width, m_TargetTexture.height), 0, 0);
m_TargetTexture2D.Apply();

var col = m_TargetTexture2D.GetPixels32();

var handle = default(GCHandle);
try
{
	handle = GCHandle.Alloc(col, GCHandleType.Pinned);
	var ptr = handle.AddrOfPinnedObject();
	libglutil_SetPixels(ptr.ToInt32(), m_TargetTexture.width, m_TargetTexture.height);
}
finally
{
	if (handle != default(GCHandle))
		handle.Free();
}

On the iPhone side, Unity's GetPixels32 is reasonably fast, so I think it's okay to get the color information normally on the Unity side and pass the pointer to the Native side. In the case of RenderTexture, it is necessary to move it to Texture2D once, and it may be necessary to be careful because that area becomes overhead.

CameraOnPost.cs


m_NativePlugin.CallStatic("sendRgbaFrame", m_Context, m_TargetTexture.GetNativeTexturePtr().ToInt32(), m_TargetTexture.width, m_TargetTexture.height);
GL.InvalidateState();

Unfortunately, Android does not perform well with Unity's ReadPixels. And the delivery of byte [] to Java is very heavy. It's too heavy to use at this stage. So on Android, the texture is passed to Java as it is, and ReadPixels is performed on the Java side. By the way, you can deliver it to NDK if you make it like you did on your iPhone. It seems that it is faster to transfer a large Byte array to Java via C # → NDK → JNI.

On the Java side, use the texture you received, draw the texture in the frame buffer with OpenGL ES, and read it with glReadPixels. But Android's glReadPixels are very heavy. It's heavy after all. Using PBO (Pixel Buffer Object) makes it a little faster (and sometimes slower), but it's still heavy. What's more, PBO requires OpenGL ES 3.0, which isn't complete in Java, so you can step on it or kick it. (You need to raise the minimum SDK API to 24 to complete Java. Can you raise it or stupid)

In order to save the acquired RGBA information in MP4 etc., it is necessary to further convert it to YUV, and when it comes to this stage, it is inevitable that it will be in a jerky state.

It may not be very useful for the hard work, but adjusting the size of the RenderTexture will improve performance to some extent.

Besides, Android had a lot of trouble. I also had a hard time finding a way to include other created so in aar. When the 64-bit version of the so file was included, it took a long time to resolve the problem that a pop-up appeared and ended the moment the application created with Unity was started.

In creating this sample, I borrowed a lot of information from our predecessors. And I saw a lot of Questions (corpses) without Answers. Everyone is in trouble with Android.

Thank you very much.

Recommended Posts

How to pass image pixel information natively from Unity
How to deploy to Heroku from a local docker image
How to migrate from JUnit4 to JUnit5
How to get the longest information from Twitter as of 12/12/2016
[IOS14] How to get Data type image data directly from PHPickerViewController?
[Rails 5] How to use gem gon ~ How to pass variables from Rails to JS ~
How to push from Tarminal to GitHub
[rails] How to display db information
How to pass Oracle Java Silver
[Swift5] How to communicate from ViewController to Model and pass a value
How to distinguish ubuntu cloud image
How to change from HTML to Haml
How to crop an image with libGDX
[Rails] How to convert from erb to haml
How to blur an image (super easy)
[IOS] How to get data from DynamoDB
How to call Swift 5.3 code from Objective-C
[Flutter] How to use C / C ++ from Dart?
Java: How to send values from Servlet to Servlet
How to pass the value to another screen
How to implement image posting using rails
[Struts] How to pass values between jsps
[Spring MVC] How to pass path variables
How to add application version information to Sentry information
[Ruby] How to convert from lowercase to uppercase and from uppercase to lowercase
How to link Rails6 Vue (from environment construction)
How to get a heapdump from a Docker container
How to dump from database (DB) to seeds file
How to make Spring Boot Docker Image smaller
How to create a class that inherits class information
How to get Class from Element in Java
How to make Unity Native Plugin (Android version)
How to give your image to someone with docker
[Java] How to switch from open jdk to oracle jdk
How to use scope and pass processing (Jakarta)
How to get SIMD optimization from HotSpot JavaVM
[ruby] How to receive values from standard input?