How to play an MP3 file via JavaScript AudioWorklet?

Question

I've followed this example and created a custom AudioWorkletProcessor which works as expected. What I'd like to do now is to stream MP3 audio from my server (I'm currently using Python/Flask) into it.

So, for example

const response = await fetch(url);
const reader = response.body.getReader();

while (true) {
  const {value, done} = await reader.read();
  if (done) break;
  // do something with value
}

which gives me a Uint8Array. How do I pass its content to the AudioWorklet instead of the current channel[i] = Math.random() * 2 - 1;?

Thank you :)

Answer 1

Firstly, MP3 is a compressed audio file format but the Web Audio API nodes only work with uncompressed sample data. You'll need to use the decodeAudioData() method of the AudioContext object to convert the bytes of the MP3 file into an AudioBuffer object.

Secondly, decodeAudioData() isn't really designed for streaming but because you're using MP3 you're in luck. See Encoding fails when I fetch audio content partially for more information.

Thirdly, the AudioContext object isn't accessible from inside an AudioWorkletProcessor, so you'll have to call decodeAudioData() from the main thread and then pass the decompressed data from your AudioWorkletNode to your AudioWorkletProcessor using their respective message ports, which are accessible from the port property of each object.

Fourthly, AudioBuffer isn't one of the allowed types that can be sent through a message port using postMessage(). Fortunately the Float32Array returned by the buffer's getChannelData() method is one of the supported types.

I'm not sure what your reason is for using an audio worklet. Depends on what you want to do with the MP3 but if all you want to do is play it then there are simpler solutions that involve lower CPU usage.

Answer 2

for (let i=0; i < value.length; i++) {
    channel[i] = value[i];
}

Answer 3

My approach to streaming audio would be to start with an <audio> tag or Audio object (same thing). That way the browser would handle all the streaming and decoding issues for me without further intervention. Then if I wanted to transfer the audio into the Web Audio API to do some client side real-time post processing then I'd use a MediaElementAudioSourceNode to achieve that. Then I'd use the built-in audio node types like BiquadFilterNode for EQ, DynamicsCompressorNode for dynamic range compression, and ConvolverNode for reverb. Only if I needed to do something that it wasn't possible to construct using the built-in node types no matter what combination they were assembled in, only then would I start writing an audio worklet. And unfortunately there are a few common things that the built-in Web Audio nodes cannot do. A single pole filter with a variable cutoff is one example (though if a fixed cutoff is acceptable then an IIRFilterNode can be used). Sample and hold is another. Being able to work around these kinds of limitations is why audio worklets rock and are super useful as small components within a larger system of nodes.

A second thing I wanted to clarify with respect to my previous answer is that although my suggestion isn't wrong per se (that is how you'd get any kind data into an AudioWorkletProcessor that isn't an input or an AudioParam, for example to emulate something similar to the buffer property on the ConvolverNode), your approach using fetch() and the phrase, "Pass its content to the AudioWorklet" (i.e. the Uint8Array) led me down a dubious line of thinking. What you have is a stream of data. The fact that the server sends it as MP3 is irrelevant. The Web Audio API works using a streams paradigm too. The more common ways to get data into an audio worklet are either:

1. To give the worklet an input. Then you can pass an instance of the worklet as a parameter to another node's connect() method, or,

2. By giving the worklet an AudioParam.

Which one is appropriate depends on what you will be doing with the content of the MP3 but if you don't already have another input defined then it's probably an input. To get the stream of MP3 data into the format the Web Audio API uses for its streams you need a MediaElementAudioSourceNode (or an AudioBufferSourceNode for a non-streaming MP3 file). Then connect the MediaElementAudioSourceNode into your worklet using connect(). Inputs are all interchangeable so if implemented this way then your worklet would be able to process any kind of audio data connected into it and wouldn't be limited to only processing MP3. When you're writing worklet code you usually don't care about where the audio is coming from. All inputs and AudioParams are just streams of samples (without compression).

This Chrome Developers blog post covers how to make worklets that have their own inputs and AudioParams.