How to access virtual directories when using OWIN handler for static files?

Question

I have this in my web.config:

<handlers>
  <remove name="StaticFile" />
  <add name="Owin" verb="" path="*" type="Microsoft.Owin.Host.SystemWeb.OwinHttpHandler, Microsoft.Owin.Host.SystemWeb" />
</handlers>

I am using OWIN so I can see what folder the user is requesting resources from and, for certain folders, I check their permissions to see if they have proper access. This was working great, but appears to have a very bad side effect. Apparently it does not support Virtual Directories that I've setup in IIS.

Is there a configuration setting I can change to fix this, or is there a method I need to override to handle the mapping myself?

Currently I just get 404 errors for anything in a virtual directory.