Problem with Media toggle in MediaStream APIs

Question

I'm trying to develop a video-conferencing application like GoogleMeet and I'm facing a problem as follows:
Let's say I'm sharing my screen with the others and now I want to turn on my camera. When I turn it on, I have to replace my video.srcObject from DisplayMedia to UserMedia and it works just fine. But when I turn my camera off, my screen should switch back to the screen that was initially shared.
For this I tried to save my DisplayMedia object first before switching, but it doesn't work that way and my screen goes blank.
for now I'm just creating a new object whenever I switch, It is good if I have to switch back to the camera but when I've to switch back to the screen as mentioned in above example, It requests the user again for which screen to share which is annoying.
Here is the code for my Camera:

const videoRef = useRef();
const userCameraCapture = async cameraIsOn => {
        if(cameraIsOn){
            if(screenStream)   
                removeVideoTracks();
            let stream = await navigator.mediaDevices.getUserMedia({video:true});
            setCameraStream(stream);
            videoRef.current.srcObject = stream;
        }else{
            removeVideoTracks();
            setCameraStream(null);
            if(screenStream){
                videoRef.current.srcObject = await navigator.mediaDevices.getDisplayMedia({video:true});
            }else{
                videoRef.current.srcObject = null;
            }
        }
    }

and for screen sharing:

const userCameraCapture = async cameraIsOn => {
        if(cameraIsOn){
            if(screenStream)   
                removeVideoTracks();
            let stream = await navigator.mediaDevices.getUserMedia({video:true});
            setCameraStream(stream);
            videoRef.current.srcObject = stream;
        }else{
            removeVideoTracks();
            setCameraStream(null);
            if(screenStream){
                videoRef.current.srcObject = await navigator.mediaDevices.getDisplayMedia({video:true});
            }else{
                videoRef.current.srcObject = null;
            }
        }
    }

These functions are toggled by a child component and they are working properly other than the problem above.
The videoRef is reference to a video tag.