How to use jq between two httpie calls to filter a stream of JSON objects

225 Views Asked by At

Here is a minimum example of how to catch the output of the Twitter API Filtered Stream and to send it to another API, in this case a FastAPI project, using httpie according to an example use case in the docs:

http --stream get \
    "https://api.twitter.com/2/tweets/search/stream" \
    "Authorization: Bearer $BEARER_TOKEN" |
    while read -r line; do
        echo "$line" | http post localhost:8888/posts
    done

This works fine, but now I am trying to alter the output of the Twitter stream using the command line JSON processor jq:

http --stream get \
    "https://api.twitter.com/2/tweets/search/stream" \
    "Authorization: Bearer $BEARER_TOKEN" |
    jq --compact-output --monochrome-output '. | {id: .data.id, rules: .matching_rules}' |
    while read -r line; do
        echo "$line" | http post localhost:8888/posts
    done

This does not work, unfortunately. It does work while doing http … | jq … but it stops working after adding | while … so even when omitting the following | http … this does not output anything.

So I assume that jq somehow breaks that streaming behavior, even when using --compact-output to have the entire JSON string in one line and --monochrome-output to prevent any color issues with the shell. Why is that? How can I fix this?

How could I filter the Twitter stream otherwise?

0

There are 0 best solutions below