Align point clouds based on GPS values

Question

I have a single layer Lidar scanner mounted on the back of a vehicle, looking down on the ground. There is also GPS sensor installed near the Lidar but the exact transformation between Lidar and GPS sensor is not known. We can imagine that the vehicle is only moving in a straight line (almost).

The Lidar gives and array of point clouds of shape nxmx3 (pc), where n is the number of frames, m is number of points per frame, and the last dim represents (x, y, z) value for each point. x is the driving dimension, y is the right of the vehicle and z is looking down on the ground. Each frame comes with a timestamp.

For each timestamp we have a GPS value (lat, long, alt) as well.

I consider (0, 0, 0) in the first frame as the origin and align all point cloud frames accordingly. Currently, I am calculating the geodesic distance between two consecutive GPS values to update the x value of the pc (assumption that any motion is only in x direction):

from geopy.distance import geodesic

# Iterate through the array of GPS coordinates
for i in range(1, len(gps)):
    # Get the current and next GPS coordinates
    current_point = gps[i]
    prev_point = gps[i - 1]
    
    # Calculate the geodesic distance between the current and prev points
    distance = geodesic(current_point, prev_point).meters
    
    # Add the distance to x value of the point cloud
    pc[i, :, 0] += distance

As you can imagine, this is not very accurate as a vehicle may not be moving exactly in a straight line.

Is there a way I can directly use GPS coordinates to update x and perhaps y coordinates of point clouds in the coordinate system as defined above?

P.S. I also have IMU data if that could be useful, but I really dont want to go the route of ICP based alignment.