Allow Pan but Restrict Zoom to Ctrl/CMd + Scroll in D3

Question

D3's zoom() functionality handles both zooming and panning. I'm looking to differentiate between the two: While restricting the user to only allow zooming when coupled with pressing the ctrl/command button, panning should always be allowed.

I've managed to block the entire zoom() behaviour using the filter method:

  function filter(event) {
    return event.ctrlKey || event.metaKey;
  }

  let defaultFilter = filter;

  let zoom = d3.zoom().filter(defaultFilter).on("zoom", handleZoom).scaleExtent(zoomRange); 

  function handleZoom(
    d3.selectAll("svg g").attr("transform", e.transform);
  }

  const svg = d3
     .attr("width", width)
     .attr("height", height)
     .call(zoom)

This then also blocks panning though. It seems I need a function which is always allowed to handle panning, and perhaps an additional function which is filtered and handles zooming?

Thanks for any ideas.