Insert \usepackage statements before hyperref, when exporting to Latex in org mode, on a file level

Question

When exporting to Latex I want to include \usepackage statements in a specific org file.

This is possible with #+LATEX_HEADER: or #+LATEX_HEADER_EXTRA:. The problem is that the packages I want to include need to be inserted before \usepackage{hyperref} as the documentation for hyperref stated.

There is the possiblity to modify the structure of the resulting latex header with the org-latex-classes variable, but this would change the behavior globaly and not just for the current org file where I want to include the package. If possible I just want to insert a package before hyperref and keep using the standard #+LATEX_CLASS: from org mode.

Here is a sample .org file I want to export with \usepackage{lmodern} as an example:

#+LATEX_CLASS: article
#+LATEX_CLASS_OPTIONS: [a4paper]
#+LATEX_HEADER: \usepackage{lmodern}

* Headline 1
  some text
* Headline 2
  some more text

The resulting Latex output with \usepackage{lmodern} after \usepackage{hyperref}, where hyperref should be the last package loaded:

% Created 2023-11-17 Fr 16:15
% Intended LaTeX compiler: pdflatex
\documentclass[a4paper]{article}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{graphicx}
\usepackage{longtable}
\usepackage{wrapfig}
\usepackage{rotating}
\usepackage[normalem]{ulem}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{capt-of}
\usepackage{hyperref}
\usepackage{lmodern}
\author{me}
\date{\today}
\title{}
\hypersetup{
 pdfauthor={me},
 pdftitle={},
 pdfkeywords={},
 pdfsubject={},
 pdfcreator={Emacs 29.1 (Org mode 9.6.6)}, 
 pdflang={English}}
\begin{document}

\tableofcontents


\section{Headline 1}
\label{sec:org1af97b4}
some text
\section{Headline 2}
\label{sec:org14cd1e2}
some more text
\end{document}