How to use GStreamers switchbin element?

Question

I have two working pipelines - one for MJPEG streams and one for H.264.

udpsrc port=5021 caps="application/x-rtp,encoding-name=JPEG,payload=26" ! \
    rtpjitterbuffer ! rtpjpegdepay ! jpegparse ! jpegdec ! autovideosink

udpsrc port=5021 caps="application/x-rtp,encoding-name=H264,payload=96" ! \
    rtpjitterbuffer ! rtph264depay ! h264parse ! openh264dec ! autovideosink

Now I don't want to switch between those manually depending on which encoding is currenty used, so I tried to use switchbin, which just doesn't seem to work for me...

What I've tried:

udpsrc port=5021 ! rtpjitterbuffer ! \
    switchbin num-paths=2 \
    path0::element="rtpjpegdepay ! jpegparse ! jpegdec" \
    path0::caps="application/x-rtp,encoding-name=JPEG,payload=26" \
    path1::element="rtph264depay ! h264parse ! openh264dec" \
    path1::caps="application/x-rtp,encoding-name=H264,payload=96" ! autovideosink

This is just not showing anything after Setting pipeline to PLAYING ....

udpsrc port=5021 ! \
    switchbin num-paths=2 \
    path0::element="rtpjitterbuffer ! rtpjpegdepay ! jpegparse ! jpegdec" \
    path0::caps="application/x-rtp,encoding-name=JPEG,payload=26" \
    path1::element="rtpjitterbuffer ! rtph264depay ! h264parse ! openh264dec" \
    path1::caps="application/x-rtp,encoding-name=H264,payload=96" ! autovideosink

This is throwing following error:

ERROR: from element /GstPipeline:pipeline0/GstUDPSrc:udpsrc0: Internal data stream error.

What am I doing wrong here?

Fyi: I am using Windows 11 23H2 with GStreamer 1.22.7 (MSVC 64-bit).

UPDATE:

Using a decodebin like recommended by Florian in this answer does not work.

Missing element: application/x-rtp decoder
ERROR: from element /GstPipeline:pipeline0/GstDecodeBin:decodebin0: Your GStreamer installation is missing a plug-in.
Additional debug info:
../gst/playback/gstdecodebin2.c(4705): gst_decode_bin_expose (): /GstPipeline:pipeline0/GstDecodeBin:decodebin0:
no suitable plugins found:
Missing decoder: application/x-rtp (application/x-rtp, encoding-name=(string)H264, payload=(int)96)

Answer 1

I don't think that you can use direct RTP/UDP streaming for your case. udpsrc cannot guess its src caps (that you explicitly set in your working pipelines).

You would use RTSP for that. With gstreamer you can set up a RTSP server very easily. You would have to install libgstrtspserver-1.0-0, and then build the sample application test-launch. Be sure to check out the version for your gst install. Once built, you can use:

# RTP/JPG
test-launch.exe "videotestsrc ! jpegenc ! rtpjpegpay name=pay0"

# RTP/H264
test-launch.exe "videotestsrc pattern=ball ! videoconvert ! x264enc ! h264parse ! rtph264pay name=pay0"

# RTP/H265
test-launch.exe "videotestsrc ! videoconvert ! x265enc ! h265parse ! rtph265pay name=pay0"

The server should listen for clients on port 8554 by default.

Now on receiver side, maybe these simple commands would be enough:

gst-play-1.0 rtsp://<SERVER_IP>:8554/test

gst-launch-1.0 playbin uri=rtsp://<SERVER_IP>:8554/test

If you want to have more control, you may then use switchbin such as:

gst-launch-1.0 rtspsrc location=rtsp://<SERVER_IP>:8554/test ! switchbin num-paths=3\
 path0::caps="application/x-rtp,encoding-name=JPEG,clock-rate=90000,payload=26" path0::element="rtpjpegdepay ! jpegdec ! videoconvert"\
 path1::caps="application/x-rtp,encoding-name=H264,clock-rate=90000,payload=96" path1::element="rtph264depay ! h264parse ! openh264dec ! videoconvert"\
 path2::caps="ANY" path2::element="fakesink   videotestsrc pattern=snow ! videoconvert" \ 
! autovideosink

If no RTPH264 nor RTPJPG is found (such as streaming H265), the fallback case would just show the snow pattern.

Answer 2

The caps in your initial pipelines are not provided by udpsink, they are there so you can tell the downstream elements what kind of data to expect. Therefore, switchbin never sees any of the caps directly and can't choose a pipeline.

I think there's two potential solutions:

1. add a decodebin element right after rtpjitterbuffer which might be able to decode RTP and content together (you'll have to try, not 100% certain)

2. if the streams have different SSRC, you could use rtpssrcdemux to direct the data to the correct pipeline. However, that requires at least a small bit of signal management code and is AFAICT not possible with gst-launch alone.