can't find media-streams from non-calling peers in multi-party video conference with socket.io and peer.js

Question

I am implementing a video conference application in MERN stack with socket.io and peer.js packages. I am having problems with multiple users let's say more than 2 peers. Let me clarify the problem first. Suppose four users (say A, B, C and D) are connected to my app. They are initialized with a socket each. Now I am creating a function that calls each and every peer that belongs in a room. If B clicks the button, it calls the function and every peer in that room gets a call from B. I have a element that wraps all the peers for that room. Also I have an object variable named peersOnConference whose key holds the id of the peer and the value holds the MediaStream object. Now if B starts call to others I get the video streams for A, B in A's UI, B, B in B's UI, B, C in C's UI and B, D in D's UI. I do not get the others' streams that do not start the call. What am I missing? Any help would be appreciated.

The bellow code is for listening the call event in useEffect.

peer?.on('call', async (call) => {
  try {
    const selfStream = await getMedia();
    setPeersOnConference((prev) => ({ ...prev, [socket.id]: selfStream }));
    call.answer(selfStream);
    setTransited(true);
    setIsAnswered(true);
    call.on('stream', (remoteStream) => {
      setPeersOnConference((prev) => ({
        ...prev,
        [call.peer]: remoteStream,
      }));
    });
  } catch (e) {
    console.log('error while receiving call');
  }
});

The function that calls to other peers:

const startGroupCall = async () => {
  try {
    const selfStream = await getMedia();
    setPeersOnConference((prev) => ({
      ...prev,
      [socket.id]: selfStream,
    }));
    for (const remotePeer in peersOnConference) {
      const call = peer.call(remotePeer, selfStream);
      call.on('stream', (remoteStream) => {
        setPeersOnConference((prev) => ({
          ...prev,
          [remotePeer]: remoteStream,
        }));
      });
    }
  } catch (e) {
    console.log('error while starting group call in catch block');
  }
};

And also I am rendering the UI using PeerVideo component for convenience. Parent div:

<div className="flex flex-wrap">
  {Object.keys(peersOnConference).map((key) => (
    <PeerVideo key={key} stream={peersOnConference[key]} />
  ))}
</div>

PeerVideo component:

import React, { useEffect, useRef } from 'react';
import { setVideoRef } from '../utils';

const PeerVideo = ({ stream }) => {
  const videoRef = useRef(null);
  useEffect(() => {
    setVideoRef(videoRef, stream);
  }, [stream]);
  return (
    <div className="w-1/2 border border-red-500">
      <video className="w-full h-full" ref={videoRef}></video>
    </div>
  );
};

export default PeerVideo;

setVideoRef utility function:

export const setVideoRef = (videoRef, stream) => {
  videoRef.current.srcObject = stream;
  let selfPlayPromise = videoRef.current.play();
  if (selfPlayPromise !== undefined) {
    selfPlayPromise
      .then((_) => console.log('playing stream'))
      .catch((e) => console.log('error while playing stream'));
  }
};