Web browsing with ARKit + SceneKit + Metal

I thought that I might want to use UIKit for virtual objects on my smartphone or AR glasses, so I tried to display UIView in the SceneKit scene. Since it is easy to check the operation, use WKWebView as the UIView.

Complete image

Reference information: ・ Tweet of operation in this Swift UI

How to display UIView in the scene

① Capture ʻUIViewand convert it toMTLTexutre ② ① is set as the material ofSCNNode` of SceneKit

only this. It works like that, but here are some addictions.

How to scroll the WebView

In order to scroll or tap the WebView that is a virtual object, it is necessary to pass the touch event to any WebView with ʻARSCNView in front. Here, I came up with a method to receive ʻUIResponder events such as touchesBegan () with ʻARSCNView and send them to touchesBegan () etc. of any WebView, but scrolling and tapping do not work at all. .. In the first place, is it possible to perform actions specific to UI parts such as scrolling or tapping a button when a UIEvent is passed? I feel like that. Lack of knowledge around here. .. .. If anyone knows, please let me know. If you google how to scroll from the program, you can see how to set it with contentOffset of ʻUIScrollView, but this should not allow you to do comfortable inertial scrolling. So, in the end, I couldn't find a good method and realized it by the following method.

・ Place WebView in the same position with the same size as ʻARSCNView -Leave ʻARSCNView in front and set ʻisUserInteractionEnabled to false -Perform a hit test and set ʻisUserInteractionEnabled of WebView in the center of the screen to true.

This allows you to scroll any WebView. However, this method has the following drawbacks.

-When scrolling, WebView cannot be set to any aspect ratio. It needs to fit the screen size. -Since the tapped position cannot be corrected to the position of WebView in the scene, event processing such as button click cannot be performed (useless).

From the above, it seems that the cases where this method can be used are limited to the cases where some information is displayed only and the operation is scroll only in the virtual object on the screen. I will try again how I can do it with Swift UI like the tweet above.

Screen capture is slow and rendering cannot keep up

The capture timing is set to renderer (_: updateAtTime :) of SCNSceneRendererDelegate, but since it is necessary to pass the screen to Capture & SceneKit in the main thread, the capture process is enqueued to the main queue in this method. .. If the screen capture is fast enough, this may be enough, but since it is a harsh process of capturing while 3D rendering while tracking with AR, the main queue rapidly expands and the screen freezes. I encountered the phenomenon. Measures to prevent the next capture until the capture is completed.

func renderer(_ renderer: SCNSceneRenderer, updateAtTime time: TimeInterval) {
    //Do not capture if the screen capture in the main queue is not finished
    //The capture process is slow, so without it, the queue will not pop and freeze.
    guard !isCaptureWaiting else { return }
    isCaptureWaiting = true

Whole source code

class ViewController: UIViewController, ARSCNViewDelegate {

    @IBOutlet weak var scnView: ARSCNView!
    @IBOutlet weak var webView1: WKWebView!
    @IBOutlet weak var webView2: WKWebView!
    @IBOutlet weak var webView3: WKWebView!
    
    private var device = MTLCreateSystemDefaultDevice()!
    private var planeNode1: SCNNode!
    private var planeNode2: SCNNode!
    private var planeNode3: SCNNode!

    private var node1Texture: MTLTexture?
    private var node2Texture: MTLTexture?
    private var node3Texture: MTLTexture?

    private var viewWidth: Int = 0
    private var viewHeight: Int = 0

    private var isCaptureWaiting = false

    override func viewDidLoad() {
        super.viewDidLoad()
        scnView.scene = SCNScene(named: "art.scnassets/sample.scn")!
        planeNode1 = scnView.scene.rootNode.childNode(withName: "plane1", recursively: true)
        planeNode2 = scnView.scene.rootNode.childNode(withName: "plane2", recursively: true)
        planeNode3 = scnView.scene.rootNode.childNode(withName: "plane3", recursively: true)
        //Stop accepting UI events once
        scnView.isUserInteractionEnabled = false
        setUIEnable(webView1: false, webView2: false, webView3: false)
        //ARSCN View is always on top
        self.view.bringSubviewToFront(scnView)
        //AR Session started
        self.scnView.delegate = self
        let configuration = ARWorldTrackingConfiguration()
        self.scnView.session.run(configuration, options: [.removeExistingAnchors, .resetTracking])
    }
    
    override func viewDidLayoutSubviews() {
        super.viewDidLayoutSubviews()
        if node1Texture == nil || node2Texture == nil || node3Texture == nil {
            viewWidth = Int(view.bounds.width)
            viewHeight = Int(view.bounds.height)
            //Allocate a texture buffer
            //WebView size= self.The size of the view
            let desc = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: .rgba8Unorm,
                                                                width: viewWidth,
                                                                height: viewHeight,
                                                                mipmapped: false)
            node1Texture = device.makeTexture(descriptor: desc)!
            node2Texture = device.makeTexture(descriptor: desc)!
            node3Texture = device.makeTexture(descriptor: desc)!
            //Site load
            webView1.load(URLRequest(url: URL(string:"https://qiita.com")!))
            webView2.load(URLRequest(url: URL(string:"https://www.apple.com")!))
            webView3.load(URLRequest(url: URL(string:"https://stackoverflow.com")!))
        }
    }
    
    //IsUserInteractionEnabled setting for each WebView
    func setUIEnable(webView1: Bool, webView2: Bool, webView3: Bool) {
        self.webView1.isUserInteractionEnabled = webView1
        self.webView2.isUserInteractionEnabled = webView2
        self.webView3.isUserInteractionEnabled = webView3
    }
    
    func renderer(_ renderer: SCNSceneRenderer, updateAtTime time: TimeInterval) {
        guard let _node1Texture = node1Texture, let _node2Texture = node2Texture, let _node3Texture = node3Texture else { return }
        //Do not capture if the screen capture in the main queue is not finished
        //The capture process is slow, so without it, the queue will not pop and freeze.
        guard !isCaptureWaiting else { return }
        isCaptureWaiting = true
        
        DispatchQueue.main.async {
            //Perform a node hit test and WebView in the center of the screen(node)Find out
            let bounds = self.scnView.bounds
            let screenCenter =  CGPoint(x: bounds.midX, y: bounds.midY)
            let options: [SCNHitTestOption: Any] = [
                .boundingBoxOnly: true,     //Tested with boundingBox
                .firstFoundOnly: true       //Returns only the foremost object
            ]
            if let hitResult = self.scnView.hitTest(screenCenter, options: options).first,
               let nodeName = hitResult.node.name {
                //Set isUserInteractionEnabled of WebView corresponding to the node in the center of the screen to true
                switch nodeName {
                case "plane1":
                    self.setUIEnable(webView1: true, webView2: false, webView3: false)
                case "plane2":
                    self.setUIEnable(webView1: false, webView2: true, webView3: false)
                case "plane3":
                    self.setUIEnable(webView1: false, webView2: false, webView3: true)
                default:
                    self.setUIEnable(webView1: false, webView2: false, webView3: false)
                }
            } else {
                self.setUIEnable(webView1: false, webView2: false, webView3: false)
            }
            
            //Update the material of the node by capturing only the WebView corresponding to the node in the center of the screen
            let setNodeMaterial: (UIView, SCNNode, MTLTexture) -> () = { captureView, node, texture in
                let material = SCNMaterial()
                material.diffuse.contents = captureView.takeTextureSnapshot(device: self.device,
                                                                            textureWidth: self.viewWidth,
                                                                            textureHeight: self.viewHeight,
                                                                            textureBuffer: texture)
                node.geometry?.materials = [material]
            }

            if self.webView1.isUserInteractionEnabled {
                setNodeMaterial(self.webView1, self.planeNode1, _node1Texture)
            } else if self.webView2.isUserInteractionEnabled {
                setNodeMaterial(self.webView2, self.planeNode2, _node2Texture)
            } else if self.webView3.isUserInteractionEnabled {
                setNodeMaterial(self.webView3, self.planeNode3, _node3Texture)
            }
            
            self.isCaptureWaiting = false
        }
    }
}

extension UIView {
    //Take a screen capture of any UIView and convert it to MTLTexture
    //Reference URL: https://stackoverflow.com/questions/61724043/render-uiview-contents-into-mtltexture
    func takeTextureSnapshot(device: MTLDevice, textureWidth: Int, textureHeight: Int, textureBuffer: MTLTexture) -> MTLTexture? {
        
        guard let context = CGContext(data: nil,
                                   width: textureWidth,
                                   height: textureHeight,
                                   bitsPerComponent: 8,
                                   bytesPerRow: 0,
                                   space: CGColorSpaceCreateDeviceRGB(),
                                   bitmapInfo: CGImageAlphaInfo.premultipliedLast.rawValue) else { return nil }
        //Avoiding upside down
        context.translateBy(x: 0, y: CGFloat(textureHeight))
        context.scaleBy(x: 1, y: -1)
        
        guard let data = context.data else { return nil }
        layer.render(in: context)
        
        textureBuffer.replace(region: MTLRegionMake2D(0, 0, textureWidth, textureHeight),
                              mipmapLevel: 0,
                              withBytes: data,
                              bytesPerRow: context.bytesPerRow)
        return textureBuffer
    }
}