how can I switch a product in 3DExperience to edit mode?
From the user interface this can simply be achieved by right-clicking the product:
In Catia V5 it was easily done with
prdRoot.ApplyWorkMode DESIGN_MODE
I cannot find anything in the docu. What is the equivalent API-method in VBA for 3DX?

You can always select the product (search by name, for example) and use CATIA.StartCommand .....