ARKit not perfectly locking in user heading

Question

I am currently playing with building an AR app to display real world landmarks using ARKit. So I know the users position and the angle at which I can see a place that is quite far away. So I set an object in that direction at let's say 5m distance.

That basically works as I want it to, but I still from time to time have occasions, where my items are just drifting off to one or the other side. That happens more frequently when e.g. when i tilt my phone a bit but also when holding my phone still.

I am now wondering if I missed some important setting?!?

What I have is the following:

let configuration = ARWorldTrackingConfiguration()
configuration.worldAlignment = .gravityAndHeading
configuration.planeDetection = [.horizontal, .vertical]

if ARWorldTrackingConfiguration.supportsSceneReconstruction(.meshWithClassification) {
    configuration.sceneReconstruction = .meshWithClassification
}

arView.session.run(configuration)

I also have a CLLocationManager that I ask to startUpdatingHeading.

Funny enough... I also display the location-managers heading and that does not drift... So I guess that this should be mainly an ARKit problem.

Is there anything else I can do, to better lock my scene to the real world?

I also noticed that sometimes my compass is just off by about 90 degrees. But not just in my app, but also e.g. the Apple Maps app. I also did not yet find a way to FORCE a recalibration?