Fancybox5 and back button multiple issues

Question

I just noticed a few bugs (some of them are features) with Fancybox5 and the back button in any browser. With the default setup, if you open the image in Fancybox and then hit the back button (or on mobile swipe) it will not just close the image and back to the page it will go to the previous page (as default behavior). However, it would be normal to just close the image and get back to the same spot. You can achieve that with the hash plugin inside Fancybox

data-slug="someslug"

And it will work fine except if you open multiple images on a page and hit the back button it will just go back to previous hashes/history. So for example, if I opened 5 images on the same page I need to click 5 times BACK button in the browser to get back to the previous page. It should work like this -> you open the image then close the image, open the new image - close the image, and then hit the BACK button and you should be back to the previous page not to all states/hashes.

The setup is the default one, fancybox loaded thru CDN and then configured

    <script>
    Fancybox.defaults.backdropClick = "close";
    Fancybox.bind("[data-fancybox]", {
       'backdropClick': 'close',
       'idle': false,
       'compact':false,
  });

</script>