Why is jumping to the same address working with "technique" A but not with B?

Question

First things first:

I have here a few lines of a hack that I wrote:

Here is the definition of a makro luaL_openlib which is actually a function pointer to 0x0090DE00:

/*
Type definitions for function signatures
*/
typedef int (luaL__openlib) (lua_State *L, const char *libname, const luaL_reg *l, int nup);


/*
Intercepting macros for function calls.
*/
#define luaL_openlib(L, libname, l, nup) ((luaL__openlib*) 0x0090DE00)(L, libname, l, nup)

This is how I am calling it:

static int luaAi_helloworld(lua_State *L)
{
    MessageBox(NULL, L"Hello World", L"", MB_OK);
    return 1;
}

static const luaL_reg ai_funcs[] = {
    { "helloworld", luaAi_helloworld },
    { NULL, NULL }
};

void open_ailib(lua_State *L)
{
    luaL_openlib(L, "ai", ai_funcs, 0);
}

The code above gets compiled into a DLL. The DLL gets loaded by my target process which .exe file I hacked a bit in order to do so.

However: This works!

The result is me being able to call ai.helloworld() in a script of a computer game in order to show the "Hello World" message box.

---

The actual thing:

Now, the code looks a bit messier but it is doing the exact same thing:

static int scse_helloworld (lua_State *L)
{
    MessageBox(NULL, L"Hello World", L"", MB_OK);
    return 1;
}

static luaL_reg scselib[] = {
    {"helloworld", scse_helloworld},
    {NULL, NULL}
};

/*
Type definitions for function signatures
*/
typedef int (luaL__openlib)(lua_State *L, const char *libname, const luaL_reg *l, int nup);

/*
Intercepting macros for function calls.
*/
#define luaL_openlib2(L, libname, l, nup) ((luaL__openlib*) 0x0090de00)(L, libname, l, nup)

SCSE_API int luaopen_scse(lua_State *L)
{
    SIZE_T numBytes;
    const int num_bytes = 16;
    unsigned char hook[num_bytes];

    HANDLE process = GetCurrentProcess();
    ReadProcessMemory(process, (LPVOID)(0x0090de00), &hook, sizeof(hook), &numBytes);
    LOGGER.LogMessage("Memory at 0x0090de00 is:\n\n");

    for (int i = 0; i < num_bytes; i++) {
        LOGGER.LogMessage("%2X ", hook[i]);
    }

    // Link lua linker functions with Supreme Commander lua functions
    if(LuaLinker::Link())
    {
        LOGGER.LogMessage("Lua linker functions successefully linked with Supreme Commander Lua functions.\n");

        LOGGER.LogMessage("luaL_openlib2()");
        luaL_openlib2(L, "scse", scselib, 0); 

        // Spoiler: This is never reached
        LOGGER.LogMessage("SCSE library successefully opened.\n");
    }
    else { 
        LOGGER.LogMessage("ERROR: Couldn't link lua linker functions with Supreme Commander Lua functions!\n");
    }

    return 1;
}

What comes additionally is me reading out the memory at 0x0090de00. I can give you the result beforehand:

Memory at 0x0090de00 is:

53 8B 5C 24 14 55 56 8B 74 24 10 57 8B 7C 24 18 

which is correct as we can see with OllyDbg - although I already knew that:

So, why are you here?

The problem is, any maybe you saw the spoiler comment above, that the log message after luaL_openlib2 is never printed. In my logfile all I see is:

Lua linker functions successefully linked with Supreme Commander Lua functions.
luaL_openlib2()

The game loads the start screen but stops functioning. Buttons are not displayed etc. It does not crash but it is dead basically. As I close it, all I get is this one last cry for help:

I am looking for an explanation - I don't understand why this is not working. All I have is a guess:

The DLL I am loading is not loaded by me. It is actually loaded by the script engine of the game. Actually, it should not be possible to load a DLL from within the game. However, since the code above is executed, the DLL is loaded. So if the DLL is loaded, maybe the memory got messed up? No, at least not the first 16 bytes of the function I am targeting so I assume, that the rest will be fine too, besides that, what should actually change that - nothing can.

I am not sure if anybody can help me here but this seems like a tough question to me.

Anybody?

Btw: Sorry for the title - feel free to make a suggestion and if you didn't try Supreme Commander yet, just get it!

Answer 1

If the game application loaded the DLL, the DLL is running in the separate process. Your application is running in the different process. In the first case the DLL and your application were running in the same process.