ToneJS PitchShift with MediaStream

Question

I'm currently building an app with pitch-shifting functionality, and I've found out that ToneJS can do that job. I'd like to know if I can extract only the pitch-shifted part of a track from a media stream.

const audioContext = new Tone.Context();
Tone.setContext(audioContext);

const mediaStreamSource = audioContext.createMediaStreamSource(
  new MediaStream([localMicrophoneTrack.getMediaStreamTrack()]),
);
const gainNode = new Tone.Gain();
const pitchShift = new Tone.PitchShift({ windowSize: 0.03, pitch: 6 });
const dstNode = audioContext.createMediaStreamDestination();

Tone.connect(mediaStreamSource, gainNode);
Tone.connect(gainNode, pitchShift);
Tone.connect(pitchShift, dstNode);

const track = dstNode.stream.getAudioTracks()[0];

When I pass the track instance further for remote users, I hear the mix of the original track and the pitch-shifted part after some delay.

Any help is appreciated. Thanks in advance.

Answer 1

If I understood your part of the code correctly, it should transfer to something like this:

const mediaStreamSource = new Tone.UserMedia();
const gainNode = new Tone.Gain(10); //you can leave this without any values, but I leave it here to make code more simple
const pitchShift = new Tone.PitchShift({ windowSize: 0.03, pitch: 6 });

mediaStreamSource.open()

mediaStreamSource.chain(gainNode, pitchShift, new Tone.Mono(), Tone.Destination);

Tone.start()

Note that Tone.UserMedia is just a wrapper around MediaStream to make it easier to work with in Tone.js (you can clearly see it here). Tone.ToneAudioNode.chain method does what you are trying to do in your code: it connects output of a node to the next node and does this for all the nodes provided. Tone.UserMedia.open method requests MediaStream and returns it as Tone.ToneAudioNode. If you have more questions, you can read the docs or feel free to ask.