Here is a minimum example of how to catch the output of the Twitter API Filtered Stream and to send it to another API, in this case a FastAPI project, using httpie according to an example use case in the docs:
http --stream get \
"https://api.twitter.com/2/tweets/search/stream" \
"Authorization: Bearer $BEARER_TOKEN" |
while read -r line; do
echo "$line" | http post localhost:8888/posts
done
This works fine, but now I am trying to alter the output of the Twitter stream using the command line JSON processor jq:
http --stream get \
"https://api.twitter.com/2/tweets/search/stream" \
"Authorization: Bearer $BEARER_TOKEN" |
jq --compact-output --monochrome-output '. | {id: .data.id, rules: .matching_rules}' |
while read -r line; do
echo "$line" | http post localhost:8888/posts
done
This does not work, unfortunately. It does work while doing http … | jq … but it stops working after adding | while … so even when omitting the following | http … this does not output anything.
So I assume that jq somehow breaks that streaming behavior, even when using --compact-output to have the entire JSON string in one line and --monochrome-output to prevent any color issues with the shell. Why is that? How can I fix this?
How could I filter the Twitter stream otherwise?