For example instead of the default operating system container around the application you could have something custom like this swing project below.

On
The host OS owns the frame decorations, but you can Create Translucent and Shaped Windows and use Frame#setUndecorated(), as shown here.
You can use a transparent image on a undecorated frame with a transparent background:
The mouse will only respond to non-opaque pixels in whatever image you use.