how to dynamically import class or module using egg or wheel files in aws

Question

I am facing an issue with importing dynamic module using egg/whl file

description : I am creating a generic module that does machine learning data prep step.

1. I have a main.py file that accepts the data_prep module name as a parameter from param file. The main.py will accept the data_prep module name and dynamic import the data_prep module of the corresponding ML model.

2. I am using python shell in aws glue job . As per aws i can import the external file using egg/whl files .

3. I created egg/whl file but didnt find a resource to dynamically import modules using egg file.

main.py (without egg file works fine in local, but cannot be used in aws python shell)

module_nm = "fraud_model_data_prep"
cls_nm = "fraud_model_data_prep"


class Dimport:
    def __init__(self, module_name, class_name):
        # __import__ method used
        # to fetch module 
        module = __import__(module_name)

        # getting attribute by 
        # getattr() method 
        my_class = getattr(module, class_name)
        my_class.main_func()

    # Driver Code


obj = Dimport(module_nm, cls_nm)

main.py (with egg file not working):

module_nm = "training_module.fraud_model_data_prep"
cls_nm = "fraud_model_data_prep"


class Dimport:
    def __init__(self, module_name, class_name):
        # __import__ method used
        # to fetch module
        module = __import__(module_name)

        # getting attribute by
        # getattr() method
        my_class = getattr(module, class_name)
        my_class.main_func()

    # Driver Code


obj = Dimport(module_nm, cls_nm)

error :

Traceback (most recent call last):
  File "C:\Users\xyz\PycharmProjects\pythonProject\main1.py", line 19, in <module>
    obj = Dimport(module_nm, cls_nm)
  File "C:\Users\xyz\PycharmProjects\pythonProject\main1.py", line 9, in __init__
    module = __import__(module_name)
ModuleNotFoundError: No module named 'fraud_model_data_prep'