I am trying to transfer a live broadcast from an IP camera or any other broadcast coming from an RTP/RTSP source to my REACT application. BUT MUST BE LIVE
My setup at the moment is:
IP Camera -> (RTP) -> FFmpeg -> (udp) -> Server(nodeJs) -> (WebRTC) -> React app
In the current situation, There is almost no delay, but there are some things here that I can't avoid and I can't understand why, and here is my question:
1) First, is the SETUP even correct and this is the only way to Stream RTP video in Web app?
2) Is it possible to avoid re-encode the stream , RTP transmission necessarily comes in H.264, hence I don't really need to execute the following command:
return spawn('ffmpeg', [
'-re', // Read input at its native frame rate Important for live-streaming
'-probesize', '32', // Set probing size to 32 bytes (32 is minimum)
'-analyzeduration', '1000000', // An input duration of 1 second
'-c:v', 'h264', // Video codec of input video
'-i', 'rtp://238.0.0.2:48888', // Input stream URL
'-map', '0:v?', // Select video from input stream
'-c:v', 'libx264', // Video codec of output stream
'-preset', 'ultrafast', // Faster encoding for lower latency
'-tune', 'zerolatency', // Optimize for zero latency
// '-s', '768x480', // Adjust the resolution (experiment with values)
'-f', 'rtp', `rtp://127.0.0.1:${udpPort}` // Output stream URL
]);
As you can se in this command I re-encode to libx264, But if I set FFMPEG a parameter '-c:v' :'copy' instead of '-c:v', 'libx264' then FFMPEG throw an error says: that it doesn't know how to encode h264 and only knows what is libx264-> Basically, I want to stop the re-encode because there is really no need for it, because the stream is already encoded to H264. Are there certain recommendations that can be made?
3) I thought about giving up the FFMPEG completely, but the RTP packets arrive at a size of 1200+ BYTES when WEBRTC is limited to up to 1280 BYTE. Is there a way to manage these sabotages without damaging the video and is it to enter this world? I guess there is the whole story with the JITTER BUFFER here
This is my server side code (THIS IS JUST A TEST CODE)
import {
MediaStreamTrack,
randomPort,
RTCPeerConnection,
RTCRtpCodecParameters,
RtpPacket,
} from 'werift'
import {Server} from "ws";
import {createSocket} from "dgram";
import {spawn} from "child_process";
import LoggerFactory from "./logger/loggerFactory";
//
const log = LoggerFactory.getLogger('ServerMedia')
// Websocket server -> WebRTC
const serverPort = 8888
const server = new Server({port: serverPort});
log.info(`Server Media start om port: ${serverPort}`);
// UDP server -> ffmpeg
const udpPort = 48888
const udp = createSocket("udp4");
// udp.bind(udpPort, () => {
// udp.addMembership("238.0.0.2");
// })
udp.bind(udpPort)
log.info(`UDP port: ${udpPort}`)
const createFFmpegProcess = () => {
log.info(`Start ffmpeg process`)
return spawn('ffmpeg', [
'-re', // Read input at its native frame rate Important for live-streaming
'-probesize', '32', // Set probing size to 32 bytes (32 is minimum)
'-analyzeduration', '1000000', // An input duration of 1 second
'-c:v', 'h264', // Video codec of input video
'-i', 'rtp://238.0.0.2:48888', // Input stream URL
'-map', '0:v?', // Select video from input stream
'-c:v', 'libx264', // Video codec of output stream
'-preset', 'ultrafast', // Faster encoding for lower latency
'-tune', 'zerolatency', // Optimize for zero latency
// '-s', '768x480', // Adjust the resolution (experiment with values)
'-f', 'rtp', `rtp://127.0.0.1:${udpPort}` // Output stream URL
]);
}
let ffmpegProcess = createFFmpegProcess();
const attachFFmpegListeners = () => {
// Capture standard output and print it
ffmpegProcess.stdout.on('data', (data) => {
log.info(`FFMPEG process stdout: ${data}`);
});
// Capture standard error and print it
ffmpegProcess.stderr.on('data', (data) => {
console.error(`ffmpeg stderr: ${data}`);
});
// Listen for the exit event
ffmpegProcess.on('exit', (code, signal) => {
if (code !== null) {
log.info(`ffmpeg process exited with code ${code}`);
} else if (signal !== null) {
log.info(`ffmpeg process killed with signal ${signal}`);
}
});
};
attachFFmpegListeners();
server.on("connection", async (socket) => {
const payloadType = 96; // It is a numerical value that is assigned to each codec in the SDP offer/answer exchange -> for H264
// Create a peer connection with the codec parameters set in advance.
const pc = new RTCPeerConnection({
codecs: {
audio: [],
video: [
new RTCRtpCodecParameters({
mimeType: "video/H264",
clockRate: 90000, // 90000 is the default value for H264
payloadType: payloadType,
}),
],
},
});
const track = new MediaStreamTrack({kind: "video"});
udp.on("message", (data) => {
console.log(data)
const rtp = RtpPacket.deSerialize(data);
rtp.header.payloadType = payloadType;
track.writeRtp(rtp);
});
udp.on("error", (err) => {
console.log(err)
});
udp.on("close", () => {
console.log("close")
});
pc.addTransceiver(track, {direction: "sendonly"});
await pc.setLocalDescription(await pc.createOffer());
const sdp = JSON.stringify(pc.localDescription);
socket.send(sdp);
socket.on("message", (data: any) => {
if (data.toString() === 'resetFFMPEG') {
ffmpegProcess.kill('SIGINT');
log.info(`FFMPEG process killed`)
setTimeout(() => {
ffmpegProcess = createFFmpegProcess();
attachFFmpegListeners();
}, 5000)
} else {
pc.setRemoteDescription(JSON.parse(data));
}
});
});
And this fronted:
<!DOCTYPE html>
<html lang="ja">
<head>
<meta charset="UTF-8" />
<title>Answer</title>
<script
crossorigin
src="https://unpkg.com/react@16/umd/react.development.js"
></script>
<script
crossorigin
src="https://unpkg.com/react-dom@16/umd/react-dom.development.js"
></script>
<script
crossorigin
src="https://cdnjs.cloudflare.com/ajax/libs/babel-core/5.8.34/browser.min.js"
></script>
<script src="https://cdn.jsdelivr.net/npm/[email protected]/runtime.min.js"></script>
</head>
<body>
<div class="main">
<div class="section" id="app1"></div>
</div>
<script type="text/babel">
let rtc;
const App = () => {
const [log, setLog] = React.useState([]);
const videoRef = React.useRef();
const socket = new WebSocket("ws://localhost:8888");
const [peer, setPeer] = React.useState(null); // Add state to keep track of the peer connection
React.useEffect(() => {
(async () => {
await new Promise((r) => (socket.onopen = r));
console.log("open websocket");
const handleOffer = async (offer) => {
console.log("new offer", offer.sdp);
const updatedPeer = new RTCPeerConnection({
iceServers: [],
sdpSemantics: "unified-plan",
});
updatedPeer.onicecandidate = ({ candidate }) => {
if (!candidate) {
const sdp = JSON.stringify(updatedPeer.localDescription);
console.log(sdp);
socket.send(sdp);
}
};
updatedPeer.oniceconnectionstatechange = () => {
console.log(
"oniceconnectionstatechange",
updatedPeer.iceConnectionState
);
};
updatedPeer.ontrack = (e) => {
console.log("ontrack", e);
videoRef.current.srcObject = e.streams[0];
};
await updatedPeer.setRemoteDescription(offer);
const answer = await updatedPeer.createAnswer();
await updatedPeer.setLocalDescription(answer);
setPeer(updatedPeer);
};
socket.onmessage = (ev) => {
const data = JSON.parse(ev.data);
if (data.type === "offer") {
handleOffer(data);
} else if (data.type === "resetFFMPEG") {
// Handle the resetFFMPEG message
console.log("FFmpeg reset requested");
}
};
})();
}, []); // Added socket as a dependency to the useEffect hook
const sendRequestToResetFFmpeg = () => {
socket.send("resetFFMPEG");
};
return (
<div>
Video:
<video ref={videoRef} autoPlay muted />
<button onClick={() => sendRequestToResetFFmpeg()}>Reset FFMPEG</button>
</div>
);
};
ReactDOM.render(<App />, document.getElementById("app1"));
</script>
</body>
</html>