I'm building a video-calling web-app, like Zoom sort of, using Agora, . The thing is that we are loading the snap filters from snap-web-sdk. Whenever a user applies a lense to their face, it should be shared with all users in the room. I am successfully applying the filter to local user but it is not shared with others.
This is my code for creating custom videoTrack with Agora and publishing that stream (and it works good)
const test = async () => {
const constraints = (window.constraints = { audio: false, video: true });
const stream = await navigator.mediaDevices.getUserMedia(constraints);
const videoTracks = stream.getVideoTracks();
const customSource = AgoraRTC.createCustomVideoTrack({
mediaStreamTrack: videoTracks[0],
});
setCustom(customSource);
await client.publish([tracks[0], customSource]);
};
and the code where I apply the lense to a canvas where the user camera is loaded:
const apiToken = "";
const cameraKit = await bootstrapCameraKit({ apiToken });
const canvas = document.getElementById("my-canvas");
const session = await cameraKit.createSession({
liveRenderTarget: canvas,
});
const agoraVideoTrack = tracks[1];
const modifiedMediaStream = new MediaStream();
modifiedMediaStream.addTrack(agoraVideoTrack.getMediaStreamTrack());
session.setSource(modifiedMediaStream);
const lens = await cameraKit.lensRepository.loadLens(
"lense_id",
"groud_id"
);
await session.applyLens(lens);
await session.play();
So I need a way to kind of combine these two and share the session as MediaStreamTrack instead of applying it to a canvas.
I tried to create some sort of div which has the canvas as a child and then share that div as MediaStreamTrack but no luck since the video would freeze and the shared frames would not be updated.