Twitter Streaming API: 'Connection broken: IncompleteRead'

Question

I am streaming some data from Twitter via the Streaming API on Python and Tweepy. The code can be seen below:

class TwitterStreamer():

    def __init__(self):
        pass

    def stream_tweets(self, twitter_data_title, key_words):
        # This handles Twitter authetification and the connection to Twitter Streaming API
        listener = StreamListener(twitter_data_title)
        auth = tweepy.OAuthHandler(api_key, api_secret_key)
        auth.set_access_token(access_token, access_secret_token)
        stream = tweepy.Stream(auth, listener)

        # This line filter Twitter Streams to capture data by the keywords: 
        stream.filter(track=key_words)

class StreamListener(tweepy.StreamListener):

    def __init__(self, twitter_data_title):
        self.fetched_tweets_filename = twitter_data_title

    def on_data(self, data):
        try:
            print(data)
        
            with open(self.fetched_tweets_filename, 'a') as tf:
                tf.write(data)
            return True
        except BaseException as e:
            print("Error on_data %s" % str(e))
        return True
      
    def on_error(self, status):
        print(status)


if __name__ == '__main__':
    twitter_streamer = TwitterStreamer()

However, after some time I always get the error 'Connection broken: IncompleteRead'. I feel like this problem might be related to an earlier post (Twitter Streaming API - urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead), however, I do not understand how he solved the problem. How can I strip the function in a way this gets less computationally expensive?

Thank you a lot!

Answer 1

This has been an ongoing problem when using the Twitter streaming API on Tweepy (I can't confirm if this happens anywhere else) that I have also experienced.

I've spent days trying to figure out how to fix it, but there is no real solution.

What I instead ended up doing was using a program supervisor called Immortal, which will instantly restart a program if it stops. I've used it for more than a year now and have had no issues with it, it restarts my programs straight away.

After installation, you use it just by typing immortal and then adding the rest of your command, e.g.:

immortal python3 twitterbot.py

When your program inevitably raises the IncompleteRead error, Immortal will start it straight back up!