How do correctly close predictions of the Mediapipe Hand Landmark model (for web) in a react application?

73 Views Asked by At

I am using the Mediapipe Hand Landmark detection model to basically create an AR jewellery application such that when I bring my hand in the camera frame it should draw the ring image onto my finger. For model loading and prediction I've referred the mediapipe hand landmarks web guide and this medium blog post. First of all, here is my react component code RingDemo.jsx:

import { useEffect, useRef, useState } from "react";
import { FilesetResolver, HandLandmarker } from "@mediapipe/tasks-vision";
import hand_landmarker_task from "/hand_landmarker.task";
import "./App.css"
import ringSrc from "/ring6.png"

const RingDemo = () => {
    let animationFrameId
    const videoRef = useRef(null);
    const canvasRef = useRef(null);
    const handModel = useRef(undefined)

    const [loading, setLoading] = useState(true)
    const ring = useRef(null)

    const startWebcam = async () => {
        try {
            const videoOptions = {
                audio: false,
                video: {
                    height: {
                        ideal: 720
                    },
                    width: {
                        ideal: 1280
                    }
                }
            }

            const stream = await navigator.mediaDevices.getUserMedia(videoOptions);
            videoRef.current.srcObject = stream;
            videoRef.current.style.transform = "scaleX(-1)"
            const videoSettings = stream.getVideoTracks()[0].getSettings()
            canvasRef.current.width = videoSettings.width
            canvasRef.current.height = videoSettings.height
        } catch (error) {
            console.error("Error accessing webcam:", error);
        }
    };

    const loadModel = async () => {
        console.log("Loading Model...");
        try {
            const vision = await FilesetResolver.forVisionTasks("https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm");

            const handLandmarkerModel = await HandLandmarker.createFromOptions(
                vision, {
                baseOptions: {
                    modelAssetPath: hand_landmarker_task,
                    delegate: "GPU",
                },
                numHands: 1,
                runningMode: "video",
            });
            handModel.current = handLandmarkerModel
        } catch (error) {
            console.error("Error initializing hand detection:", error);
        }
    };


    const detectHands = () => {

        // Handling the last frame which gets rendered when the component is cleaned
        if (!handModel.current && animationFrameId) {
            window.cancelAnimationFrame(animationFrameId);
            animationFrameId = null
            return
        }

        if (videoRef.current && videoRef.current.readyState >= 2) {
            const detections = handModel.current.detectForVideo(videoRef.current, performance.now());
            if (detections.landmarks) {
                drawLandmarks(detections.landmarks);
            }
        }
        animationFrameId = window.requestAnimationFrame(detectHands);
    };

    const drawLandmarks = (landmarksArray) => {
        const canvas = canvasRef.current;
        const ctx = canvas.getContext('2d');
        ctx.clearRect(0, 0, canvas.width, canvas.height);
        ctx.fillStyle = 'white';

        landmarksArray.forEach(landmarks => {
            ring.current.draw(ctx, canvas, landmarks)
        });
    };

    const loadRing = () => {
        try {
            console.log("Loading Ring...");
            const ringImage = new Image()
            ringImage.src = ringSrc
            ringImage.onload = () => {
                ring.current = new Ring(ringImage)
            }
        } catch (error) {
            console.error("Error loading resources:", error);
        }
    };

    useEffect(() => {
        (async () => {
            loadRing()
            await loadModel()
            setLoading(false)
        })()

        return () => {
            if (videoRef.current && videoRef.current.srcObject) {
                videoRef.current.srcObject.getTracks().forEach(track => track.stop());
            }
            if (handModel.current) {
                handModel.current.close();
            }
        };
    }, [])

    useEffect(() => {
        (async () => {
            if (!loading) {
                if (handModel.current) {
                    await startWebcam()
                    detectHands()
                }
            }
        })()
    }, [loading])


    return (
        <div className="main">
            {loading ? <h1>Loading Please Wait</h1> : (
                <div className="video_box">
                    <video className="video_scr" ref={videoRef} autoPlay playsInline ></video>
                    <canvas ref={canvasRef} style={{ backgroundColor: "transparent" }}></canvas>
                </div>
            )}
        </div>
    );
};

export default RingDemo;

Now whenever I run the react application it runs fine and I am able to see the ring on my finger. However, as soon as I save the RingDemo.jsx file the cleanup code in the first useEffect .i.e:

useEffect(() => {
        (async () => {
            loadRing()
            await loadModel()
            setLoading(false)
        })()

        return () => {
            if (videoRef.current && videoRef.current.srcObject) {
                videoRef.current.srcObject.getTracks().forEach(track => track.stop());
            }
            if (handModel.current) {
                handModel.current.close();

            }
        };
    }, [])

This (return block) gets executed and due to which handModel.current.close(); is called. When this happens I get this log on my console:

I0229 18:53:36.093000 1927856 gl_context_webgl.cc:132] Successfully destroyed WebGL context with handle 1 vision_wasm_internal.js:9

After this the entire application tab freezes and I have no choice but to close the tab by clicking multiple times or in some cases to end the browser process itself to remove the tab.

Although if I reset the handModel ref in the cleanup function and reload the model by resetting the loading state, still I get the above mentioned log but this time everything works fine and my tab does not freeze. Like this:

useEffect(() => {
        (async () => {
            loadRing()
            await loadModel()
            setLoading(false)
        })()

        return () => {
            if (videoRef.current && videoRef.current.srcObject) {
                videoRef.current.srcObject.getTracks().forEach(track => track.stop());
            }
            if (handModel.current) {
                handModel.current.close();
                handModel.current = undefined
                setLoading(true)
            }
        };
    }, [])

Is there a correct way to close the hand landmark model's predictions? Because resetting the handModel ref is quite inefficient and I need to reload the model on every save. I am quite new to react and stackoverflow in general so, pardon my mistakes and please let me know if my question is clear or do you need any more information. Thank you very much in advance, any help would be highly appreciated.

0

There are 0 best solutions below