Even though pop over background colour is clear there is a weird shadow behind the popover view this issue is happening only in 13.1 and 13.2 and it is working fine in 13 and below
I can see in view hierarchy that UIWindow/UITransitionView/_UICutoutShadowView has image view with shadow image which is only in 13.1 but image view has empty image in 13
controller.modalPresentationStyle = .popover
controller.popoverPresentationController?.permittedArrowDirections = .up
controller.popoverPresentationController?.delegate = controller
controller.popoverPresentationController?.sourceView = sourceView
controller.popoverPresentationController?.popoverBackgroundViewClass = FilterBackgroundView.self
present(controller, animated: false)

On UI inspect there is a UIImageView of type _UICutoutShadowView which is causing it. So I managed to fix this by creating a custom UIPopoverBackgroundView and hiding this ghost view.