FFMPEG Dash with tiles of thumbnail images

Question

As of DASH-IF IOP version 4.2, section 6.2.6 defines the notion of image-based tracks in DASH: https://dashif.org/docs/DASH-IF-IOP-v4.3.pdf.

This is the ability to have an adaption set made up of mime type images that themselves are a strip of low resolution thumbnails. A player will use these thumbnails when the user hovers their mouse over the video timeline and get a preview of the the frame at that approximate timecode.

Theo player website has a page dedicated to this function for playback: https://www.theoplayer.com/blog/in-stream-thumbnail-support-dvr-dash-streams

I need to generate a dash stream (not live) using ffmpeg that also contains these thumbnails. I already have an ffmpeg command that will generate the film strip of jpgs which outputs a thumbnail every 5 seconds of input video and joins 5 of these together in a single jpg:

ffmpeg -i INPUT -q:v 20 -vf "select=not(mod(n\,125)),scale=480:270,tile=5x1" -vsync vfr output%d.jpg

and the mpeg dash itself:

ffmpeg -i INPUT -y -map 0 -acodec aac -ac 2 -ar 48000 -s 960x540 -vcodec libx264 -f dash -preset veryfast -b:v:2 1500k -seg_duration 2 output.mpd

But I cannot find a way in ffmpeg to include the thumbnails in the dash mpd file.

Answer 1

I have used this exact method to generate the tiled-thumbnails but I had to manually inject the adaptation set into the output.mpd. The BaseURL points to a static image server where the tiled images are stored and the video / audio / captions are served from a streaming server. You will need to make sure that the Adaptation set id does not clash with any of the other values in the manifest.

Here is an example of the adaptation set that I added into the DASH period.

<AdaptationSet contentType="image" id="5" mimeType="image/jpeg">
  <BaseURL>
    http://origin.yourdomain.com/storage/thumbs/video_id/
  </BaseURL>
  <SegmentTemplate duration="120" media="$RepresentationID$/number/tile$Number%03d$.jpg" startNumber="1"/>
  <Representation bandwidth="12895" height="432" id="thumbnails_256x144" width="1024">
    <EssentialProperty schemeIdUri="http://dashif.org/guidelines/thumbnail_tile" value="4x3"/>
  </Representation>
</AdaptationSet>