logstash vs a task queue for nodejs application

Question

I have a nodejs web server where there seem to be a couple of bottlenecks preventing it from fully functioning peak load.

• logging multiple events to our SQL server
• logging multiple events to our elastic cluster

under heavy load , both SQL and elastic seem to reject my requests and/or considerably slow performance. So I've decided to reduce the load on these DBs via Logstash (for elastic) and an async task queue (for SQL)

Since i'm working on limited time , i'm wondering if i can solve these issues with just an async task queue (like kue) where i can push both SQL and elastic logs.

Do i need to implement logstash as well? or does an async queue solve the same thing as logstash.

Answer 1

You can try Async library's Queue feature and try and make it run on a child process or much better in a separate server as a Microservice for queues. That way you can move the load to another location, giving you a boost on your app server.

Answer 2

As you mentioned you are using azure I would strongly recommend using their queue solution plus a couple of azure functions to handle the read from the queue and processing.

I've rolled my own solution before using node.js and rabbitmq with node workers to read from the queue and write to elastic search because the solution couldn't go into the cloud.

It works and is robust but it takes quite a bit of configuration and custom code that is a maintenance nightmare. Ripped that out as soon as I could.

The benefits of using the azure service are:

• Very little bespoke configuration is required.
• Less custom code === less bugs & maintainence
• scaling isn't an issue, some huge businesses rely on this
• no 2am support calls, if azure is down they are going to fix it... fast
• much cheaper, unless the throughput is massive and constant the scaling model is much cheaper and azure functions are perfect as you won't have running servers sitting there doing nothing when the queue is empty.

Same could be said for AWS and Google Cloud.