I need to duplicate the stdout of a producer and feed it to two consumer in a synchronized fashion.
consumer 1
producer | duplicator |
consumer 2
This can easily be accomplished for example via a dup.c program doing the job:
#include <stdio.h>
int main()
{
char *line = NULL;
size_t size;
while (getline(&line, &size, stdin) != -1) {
fprintf(stdout, "%s", line);
fprintf(stderr, "%s", line);
}
return 0;
}
and then:
(cat file.txt | ./dup | ./consumer1.py ; ) 2>&1 | ./consumer2.py
However, if consumer 1 is faster than consumer 2 we have a problem. E.g., consumer 1 is already at line 50,000 while consumer 2 is at line 17,000. If I "tail -f" the output of consumer1.py and consumer2.py we can see that because consumer2.py is slower (due to heavy processings inside)
output consumer1.py at time x:
50,546
50,547
50,548
50,549
50,550
50,551
50,552
output consumer2.py at time x:
17,315
17,316
17,317
17,318
17,319
17,320
17,321
For my system I need that both consumers are at the same line, hence the faster consumer needs to be restricted. It should be somehow possible. Any suggestions how to accomplish this? Maybe via signal handling or pipe specific voodoo? Thanks!