Erlang SSL TCP Server And Garbage Collection

Question

Edit: The issue seems to be with SSL acccpeting and a memory leak.

I have noticed if you have long lived Processes(its a server), and clients send data to the server(recv), the Erlang garbage collection never gets called (or rarely)

Servers need data (to preform actions), and the data can be variable length (due to a message like "Hello" or "How are you doing"). Because of this, it seems like the Erlang process will accumulate garbage.

How can you properly handle this, the Erlang process has to touch the recv data, so is it unavoidable? Or do you have to come up with designs that touches the variable length data the less amount of times (like immediately passing it to a port driver).

Spawning a worker to process the data is a bad solution(millions of connections ...), and using workers would basically be the same thing, right? So that leaves me with very few options.

Thanks ...

Answer 1

Just incase anyone finds themselves in the same boat. This is known/happening when hammering the server with many connections at once. I think.

Without ssl, my sessions are at ~8KB roughly, and resources are being triggered for GC as expected. With SSL its an increase to ~150KB, and the memory keeps growing and growing and growing.

http://erlang.org/pipermail/erlang-questions/2017-August/093037.html

Answer 2

If the server holds on to the received message longer than it needs to, it's a bug in the server implementation. Normally, the server should forget all or most references to the data in a request when that request has finished processing, and the data will then become garbage and will eventually get collected. But if you stick the data from each request in a list in the process state, or in an ets table or similar, you will get a memory leak.

There is a bit of an exception with binaries larger than 64 bytes, because they are handled by reference counting, and to the memory allocator it can look like there's no need to perform a collection yet, although the number of bytes used off-heap by such binaries can be quite large.

Answer 3

You might be having a problem with large SSL/TLS session tables. We (OTP team) have historically had some issues with those tables as they may grow very large if you do not have some limiting mechanisms. Alas in the latest ssl version one of the limiting mechanism was broken, however it is easily fixed by this patch. As a workaround you can also sacrifice some efficiency and disable session reuse.

diff --git a/lib/ssl/src/ssl_manager.erl b/lib/ssl/src/ssl_manager.erl
index ca9aaf4..ef7c3de 100644
--- a/lib/ssl/src/ssl_manager.erl
+++ b/lib/ssl/src/ssl_manager.erl
@@ -563,7 +563,7 @@ server_register_session(Port, Session, #state{session_cache_server_max = Max,

do_register_session(Key, Session, Max, Pid, Cache, CacheCb) ->
     try CacheCb:size(Cache) of
-   Max ->
+   Size when Size >= Max ->
    invalidate_session_cache(Pid, CacheCb, Cache);
_ ->    
    CacheCb:update(Cache, Key, Session),