1-3 'SyntaxError: Unexpected end of JSON input' errors occurring per minute when streaming tweets in Node

Question

Using express and Node.js, I'm using the twitter streaming API and the needle npm package for accessing APIs to pull tweets related to keywords. The streaming is functional and I am successfully pulling tweets using the following (simplified) code:

const needle = require('needle');
const TOKEN = // My Token 
const streamURL = 'https://api.twitter.com/2/tweets/search/stream';

function streamTweets() {

    const stream = needle.get(streamURL, {
        headers: {
            Authorization: `Bearer ${TOKEN}`
        }
    });

    stream.on('data', (data) => {
        try {
            const json = JSON.parse(data); // This line appears to be causing my error
            const text = json.data.text;
        } catch (error) {
             console.log("error");
        }
    });
}

However, no matter which search term I use (and the subsequent large or small volume of tweets coming through), the catch block will consistently log 1-3 errors per minute, which look like this:

SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at PassThrough.<anonymous> (C:\Users\danie\OneDrive\Documents\Personal-Projects\twitter-program\server.js:56:31)
    at PassThrough.emit (events.js:315:20)
    at addChunk (internal/streams/readable.js:309:12)
    at readableAddChunk (internal/streams/readable.js:284:9)
    at PassThrough.Readable.push (internal/streams/readable.js:223:10)
    at PassThrough.Transform.push (internal/streams/transform.js:166:32)
    at PassThrough.afterTransform (internal/streams/transform.js:101:10)
    at PassThrough._transform (internal/streams/passthrough.js:46:3)
    at PassThrough.Transform._read (internal/streams/transform.js:205:10).

I've seen previous advice which says that data can be fired in multiple chunks, and to push the chunks to an array i.e. something like the following:

let chunks = [];
        stream.on('data', (dataChunk) => {
chunks.push(dataChunk);
        }).on('end',() => {
// combine chunks to create JSON object
})

But this didn't work either (may have been my implementation but I don't think so) and now I'm wondering if it's perhaps an error with the twitter API, because most of the tweet objects do come through correctly. I should note that the streamTweets() function above is called from an async function, and I am also wondering if that is having something to do with it.

Has anyone else encountered this error? Or does anyone have any idea how I might be fix it? Ideally i'd like 100% of the tweets to stream correctly.

Thanks in advance!

Answer 1

For future readers, this error is triggered by Twitter's heartbeat message that is sent every 20 seconds. Per the documentation:

The endpoint provides a 20-second keep alive heartbeat (it will look like a new line character).

Adding a guard against parsing the empty string will prevent the JSON parsing error.

if (data === "")
  return

An empty string is invalid JSON, hence the emitted error.

---

Now, acknowledging that the heartbeat exists, it may be beneficial to add read_timeout = 20 * 1000 in the needle request to avoiding a stalled program with no data, be that due to a local network outage or DNS miss, etc.