Visual Studio Setup Project (vdproj) - STDOLE, how to properly register COM DLL?

Question

I am using Visual Studio 2015 Setup Project.

I have a COM DLL in setup that has a reference to the STDOLE (system type library, {00020430-0000-0000-C000-000000000046}, that must not be registered, as of my understanding, because it is a protected file/registration). If you try to register it, you'll hit an error (warning) when installing the MSI.

Nevertheless, the vdproj extension adds the the registration for this DLL to the "TypeLib" MSI table. It appears that it does not do it always though, but under some conditions I cannot deduce. I have no option to exclude that tlb, it is somehow keeps reappearing (from dependencies probably?)

1. Why does vdproj tries to register that DLL that should not be registered?
2. How to avoid this? I tried adding manual post-edit of the produced MSI with a script to remove that tlb, and that worked; however it seems to be a completely wicked approach to the problem.

For this particular project, I am stuck with the Visual Studio Installer, and cannot switch to more reasonable and transparent alternatives (such as WiX). Any help or advice is highly appreciated!

Answer 1

VB6 Runtime: The registry key: Typelib\{000204EF-0000-0000-C000-000000000046} is part of the Visual Basic 6 Runtime and it is a known issue in WiX that heat.exe extraction of COM data erronously brings along this registry key for each ActiveX or COM component. Note that this is not WiX's fault, but a general issue with VB6. Hence the problem you see in Visual Studio installer projects as well.

From the WiX help manual (for heat.exe) (multi-purpose tool, one of which is COM registration extraction): "...When registering a COM component created in VB6 it adds registry entries that are part of the VB6 runtime component. This flag is recommend for VB6 components to avoid breaking the VB6 runtime on uninstall. The following values are excluded: CLSID\{D5DE8D20-5BB8-11D1-A1E3-00A0C90F2731}, Typelib\{EA544A21-C82D-11D1-A3E4-00A0C90AEA82}, Typelib\{000204EF-0000-0000-C000-000000000046}, any interfaces that reference these two type libraries`"

Resolution: And now the hard part, what is the actual fix? Frankly I don't have the time to test right now, but I suppose you can just open the final MSI with Orca and remove these entries - should they be there. I guess that is exactly what you are doing now, so you are back to square one, but at least there is a bit of clarity as to what causes the problem. I suppose you can try to run the build when the VB6 component is unregistered as opposed to when it is registered? That would make some sort of sense that it is only extracting the keys if the component is unregistered when you build? If this is the case, maybe try to self-register your VB6 component prior to the setup build.

---

RegSpy.exe: You can use a tool called RegSpy (or RegSpy2 actually) to extract the COM registration from a COM file. You can find the exe in the "Release" folder, then you go:

regspy.exe richtx32.ocx >> richtx32.reg

Then you can import the reg file to your Visual Studio Installer project. Remember to replace the hard coded paths with MSI directory properties and remove the erroneous VB6 entries mentioned above.

VSProj: In Visual Studio, Right click Visual Studio Installer Project => View => Registry => Right Click Top Node => Import...