Calling a function with node-ffi-napi (always) leaks memory

934 Views Asked by At

(Original question involving CGO was deleted and replaced with a simpler demonstration that shows what I think to be the core problem, the old question can still be accessed in the edit history.)

Use case: I am using the npm package ffi-napi to call a native function (nothing surprising).

Problem: each call, even of a void() function, leaks memory, which seems, well, a bit problematic.

Most simple repro I came up with:

const lib = ffi.Library(null, {
    'atoi': ['int', ['string']]
});

let i = 0;
while (++i) {
    lib.atoi('1234');

    global.gc();

    if (i % 100000 == 0) {
        console.log(`iter. #${i} RSS MBs used: ${process.memoryUsage().rss / 1024 / 1024}`);
    }
}

Outputs:

iter. #100000 RSS MBs used: 304.8359375
iter. #200000 RSS MBs used: 509.48828125
iter. #300000 RSS MBs used: 755.44140625
iter. #400000 RSS MBs used: 905.8203125
iter. #500000 RSS MBs used: 1123.90234375
iter. #600000 RSS MBs used: 1370.88671875
iter. #700000 RSS MBs used: 1612.45703125
-- CUT --
iter. #6200000 RSS MBs used: 11568.64453125
iter. #6300000 RSS MBs used: 11794.2109375
iter. #6400000 RSS MBs used: 11905.625
iter. #6500000 RSS MBs used: 12026.44921875

<--- Last few GCs --->

[1217086:0x5627be68a030]   168590 ms: Mark-sweep 2003.3 (2070.7) -> 1992.3 (2072.7) MB, 3546.0 / 170.8 ms  (average mu = 0.101, current mu = 0.024) allocation failure scavenge might not succeed
[1217086:0x5627be68a030]   171579 ms: Mark-sweep 2005.6 (2072.7) -> 1994.6 (2074.7) MB, 2907.8 / 231.8 ms  (average mu = 0.068, current mu = 0.027) allocation failure scavenge might not succeed


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x7fe054be1679]
Security context: 0x146fda0db989 <JSObject>
    1: debug [0x11881d6991b9] [/home/morgan/Documents/Code/nogame/node_modules/debug/src/common.js:~64] [pc=0x553685c70a5](this=0x036c76042a69 <JSGlobal Object>)
    2: arguments adaptor frame: 1->0
    3: ./apps/server/src/main.ts [0x1cd68b8fffa9] [/home/morgan/Documents/Code/nogame/dist/apps/server/main.js:~94] [pc=0x553685d6a90](this=0x23bff30191f9 <Objec...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x7fe053f2da9c node::Abort() [/lib/x86_64-linux-gnu/libnode.so.72]
 2: 0x7fe053e61872  [/lib/x86_64-linux-gnu/libnode.so.72]
 3: 0x7fe054107aea v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/lib/x86_64-linux-gnu/libnode.so.72]
 4: 0x7fe054107dcb v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/lib/x86_64-linux-gnu/libnode.so.72]
 5: 0x7fe0542b59b9  [/lib/x86_64-linux-gnu/libnode.so.72]
 6: 0x7fe0542c5da7 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/lib/x86_64-linux-gnu/libnode.so.72]
 7: 0x7fe0542c6a8f v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/lib/x86_64-linux-gnu/libnode.so.72]
 8: 0x7fe0542c8d7c v8::internal::Heap::AllocateRawWithLightRetry(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/lib/x86_64-linux-gnu/libnode.so.72]
 9: 0x7fe0542c8de8 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/lib/x86_64-linux-gnu/libnode.so.72]
10: 0x7fe05428d7ef v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/lib/x86_64-linux-gnu/libnode.so.72]
11: 0x7fe0545df559 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/lib/x86_64-linux-gnu/libnode.so.72]
12: 0x7fe054be1679  [/lib/x86_64-linux-gnu/libnode.so.72]
[1]    1217086 IOT instruction  node --expose-gc dist/apps/server/main.js

NB: The same problem occurs with ffi-cross.

Valgrind log here for 5000 iterations: https://pastebin.com/p3Uv4WwR.

The leaking stack appears clearly:

==1223109== 400,160 bytes in 5,002 blocks are definitely lost in loss record 61 of 61
==1223109==    at 0x4A3F723: operator new(unsigned long) (vg_replace_malloc.c:417)
==1223109==    by 0x5428CE9: napi_create_reference (in /usr/lib/x86_64-linux-gnu/libnode.so.72)
==1223109==    by 0xDACDEFF: (anonymous namespace)::InstanceData::GetBufferData(napi_value__*) (in /home/morgan/Documents/Code/redacted/node_modules/ref-napi/prebuilds/linux-x64/node.napi.node)
==1223109==    by 0xDCE372F: FFI::FFI::FFICall(Napi::CallbackInfo const&) (in /home/morgan/Documents/Code/redacted/node_modules/ffi-napi/build/Release/ffi_bindings.node)
==1223109==    by 0xDCE6B0D: Napi::details::CallbackData<void (*)(Napi::CallbackInfo const&), void>::Wrapper(napi_env__*, napi_callback_info__*) (in /home/morgan/Documents/Code/redacted/node_modules/ffi-napi/build/Release/ffi_bindings.node)
==1223109==    by 0x5421127: ??? (in /usr/lib/x86_64-linux-gnu/libnode.so.72)
==1223109==    by 0x56AFB48: v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo) (in /usr/lib/x86_64-linux-gnu/libnode.so.72)
==1223109==    by 0x56AFEF4: ??? (in /usr/lib/x86_64-linux-gnu/libnode.so.72)
==1223109==    by 0x56B0749: ??? (in /usr/lib/x86_64-linux-gnu/libnode.so.72)
==1223109==    by 0x56B101C: v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) (in /usr/lib/x86_64-linux-gnu/libnode.so.72)
==1223109==    by 0x6123778: ??? (in /usr/lib/x86_64-linux-gnu/libnode.so.72)
==1223109==    by 0x60A8F03: ??? (in /usr/lib/x86_64-linux-gnu/libnode.so.72)

I will post an issue in the library's GitHub but any help would be greatly appreciated.

0

There are 0 best solutions below