MUI Date Picker how to add a new button next to the openPickerButton (calendar icon)?

Question

I would like to add a clear button (X) to MUI Date Picker next to the calendar icon.

X is on the wrong place

My problem is that I am trying to add it with using endAdornment, but then the clear icon doesn't show up, it only shows up when I use startAdornment. How can I position it next to the calendar icon?