PeriodicWorkRequestBuilder is set 40 years in the future

Question

I'm trying to set up a PeriodicWorkRequestBuilder that would download some data from our servers every morning at 5am. I have noticed that the worker runs once, and never again, so I'm wondering what is wrong with our configuration, or if there is something that messes up with our configuration. Here is the initial set up I have:

        private val uniqueWorkName = MediaDownloadWorker::class.java.simpleName

        private const val WORKER_TAG_OUTPUT = "media_download_worker"
        const val FOREGROUND_NOTIFICATION_ID = 20240212
        const val DOWNLOAD_NOTIFICATION_CHANNEL_ID = "download_message_notifications"

        /** Enqueues a new worker to download media only if not enqueued already. */
        fun enqueue(context: Context) {
            val manager = WorkManager.getInstance(context)

            val currentDate = Calendar.getInstance()
            val dueDate = Calendar.getInstance()
            // Set Execution around 05:00:00 AM.
            dueDate.set(Calendar.HOUR_OF_DAY, 5)
            dueDate.set(Calendar.MINUTE, 0)
            dueDate.set(Calendar.SECOND, 0)

            if (dueDate.before(currentDate)) {
                dueDate.add(Calendar.HOUR_OF_DAY, 24)
            }
            val timeDiff = dueDate.timeInMillis - currentDate.timeInMillis

            val constraints =
                Constraints.Builder()
                    .setRequiredNetworkType(NetworkType.UNMETERED)
                    .setRequiresBatteryNotLow(true)
                    .build()
            val dailyWorkRequest =
                PeriodicWorkRequestBuilder<MediaDownloadWorker>(1, TimeUnit.DAYS)
                    .setConstraints(constraints)
                    .setInitialDelay(timeDiff, TimeUnit.MILLISECONDS)
                    .addTag(WORKER_TAG_OUTPUT)
                    .build()

            manager.enqueueUniquePeriodicWork(
                "MediaDownloadWorker", ExistingPeriodicWorkPolicy.KEEP, dailyWorkRequest)
        }

The worker itself is an extension of CoroutineWorker:

class MediaDownloadWorker(private val context: Context, workerParameters: WorkerParameters) :
    CoroutineWorker(context, workerParameters) {

And inside the worker I use androidx.media3.exoplayer.offline.DownloadManager to schedule a download request for a list of files. I use explicitly the DownloadManager since that what we use later to display the videos, and play the voice.

val downloadRequest =
            DownloadRequest.Builder(Util.getUtf8Bytes(mediaUrl).toString(), Uri.parse(mediaUrl))
                .setMimeType(mimeType)
                .build()
        downloadManager.addDownload(downloadRequest)

To test things out (since it was not updating more than once), I removed the inital delay, and added interval to 15 mins. I created an observer that pulls the worker info, and this is what I see. Before it runs the first time, the nextScheduleTimeMillis is when it was supposed to be, a few minutes in the future (or 5 am):

Download Worker: WorkInfo{id='03a8deb5-386c-4ac4-a0fd-0be1aa78c8b7', state=ENQUEUED, outputData=Data {}, tags=[worker.MediaDownloadWorker, media_download_worker], progress=Data {}, runAttemptCount=0, generation=0, constraints=Constraints{requiredNetworkType=UNMETERED, requiresCharging=false, requiresDeviceIdle=false, requiresBatteryNotLow=true, requiresStorageNotLow=false, contentTriggerUpdateDelayMillis=-1, contentTriggerMaxDelayMillis=-1, contentUriTriggers=[], }, initialDelayMillis=0, periodicityInfo=PeriodicityInfo{repeatIntervalMillis=900000, flexIntervalMillis=900000}, nextScheduleTimeMillis=1709184279367}, stopReason=-256; next: 1709184279367

Once the worker runs, and I observe the info, the nextScheduleTimeMillis is ridiculously far in the future, although I very clearly specify the interval. Here is the output. Notice the timestamp being in 2026!!

Download Worker: WorkInfo{id='03a8deb5-386c-4ac4-a0fd-0be1aa78c8b7', state=RUNNING, outputData=Data {}, tags=[worker.MediaDownloadWorker, media_download_worker], progress=Data {}, runAttemptCount=1, generation=0, constraints=Constraints{requiredNetworkType=UNMETERED, requiresCharging=false, requiresDeviceIdle=false, requiresBatteryNotLow=true, requiresStorageNotLow=false, contentTriggerUpdateDelayMillis=-1, contentTriggerMaxDelayMillis=-1, contentUriTriggers=[], }, initialDelayMillis=0, periodicityInfo=PeriodicityInfo{repeatIntervalMillis=900000, flexIntervalMillis=900000}, nextScheduleTimeMillis=9223372036854775807}, stopReason=-256; next: 9223372036854775807

What could possibly interfere with this? We use a worker in the exactly the same way for updating our Widget, and it works like a charm. Help!