How to avoid gestureDetector from resetting initialPosition to origin after every horizontalDragStart/update?

Question

I've the following minimal code to achieve a horizontal-only movable container widget.

void main() {
  runApp(const MyApp());
}

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  @override
  Widget build(BuildContext context) {
    return const MaterialApp(
      home: Scaffold(
        body: Center(
          child: LeftHolder(),
        ),
      ),
    );
  }
}

class LeftHolder extends StatefulWidget {
  const LeftHolder({super.key});

  @override
  State<LeftHolder> createState() => _LeftHolderState();
}

class _LeftHolderState extends State<LeftHolder> {
  var prevPos = const Offset(0, 0);
  var curPos = const Offset(0, 0);

  void updatePosition(Offset newPosition) {
    setState(() {
      curPos = newPosition;
    });
  }

  @override
  Widget build(context) {
    return Transform.translate(
      offset: curPos,
      child: GestureDetector(
        onHorizontalDragStart: (details) {
          updatePosition(Offset(details.localPosition.dx, 0));
        },
        onHorizontalDragUpdate: (details) {
          updatePosition(Offset(details.localPosition.dx, 0));
          prevPos = Offset(details.localPosition.dx, 0);
        },
        onHorizontalDragEnd: (details) {
           updatePosition(prevPos);
        },
            child: Container(
              color: Colors.red,
              width: 20,
              height: 20,
            ),
          ),
        );
      }
    }

Problem :-

1. Start dragging the container, continue dragging and stop dragging. The container will move as expected.
2. If I start dragging again, the container first resets its position to the origin (which in this case is center of the screen, since the widget is Centered) and then moves. The container isn't remembering where I stopped dragging it previously.

It seems like the

details.localPosition

property resets to Offset(0,0) after every drag-end. Is this the default behaviour of GestureDetector ? And if it isn't, how can I force the container to always begin dragging from the position it was drag-suspended in the prior move ?

Note:- I've tried the closest relevant question to this, but it doesn't change the behaviour.