Why does my pitchShift with multiple samples synced with Transport sound bad and get worse over time?

Question

I'm trying to create a multi-track player where each track's volume can be altered individually (working fine), and where all tracks can can be pitch-adjusted (not separately, but the whole group).

To get a visual of what I'm aiming at, this is the rough UI after a couple of hours of work. Tracks were exported from Logic DAW:

I'm building in React and here is how the tracks are initialized:

const initAudio = () => {

  const files = [
    "Accompaniment Those Memories of You.wav",
    "Baritone Those Memories of You.wav",
    "Lead Those Memories of You.wav",
    "Tenor Those Memories of You.wav",
  ]

  tracks = files.map(
    filename => {
      let pitchShift = new Tone.PitchShift().toDestination()
      const player = new Tone.Player(
        {
          url: 'audio/' + filename,
          loop: false
        }
      ).connect(pitchShift).sync().start(0)
      
      return {
        player,
        pitchShift,
        name: filename.split('.')[0],
      }
    }
  )
}

Here is how the pitch is being changed, but when I do this, it sounds positively bad, and just gets worse over the 30 seconds of the mp3 files.

  const changePitch = (changeVal) => {
    tracks.forEach(
      track => track['pitchShift'].pitch = track['pitchShift'].pitch + changeVal
    )
  }

I don't know if this is down to which one (or other) of the following:

• Bad setup or use of connect or toDestination (I'm new)
• Inherent syncing issues with Transport
• Processor limitation
• etc

I'd really like this to work, but as it is, it's so bad it's comical.

Anything obviously wrong here?

For what it's worth, I also tried this with just one pitchShift object that all tracks were connected to. Happy to share the whole components, but figured that would be a bit much here.