Streamsets job logs to S3

Question

I am looking for solution where we can store all logs (Info, Debug etc) of a Streamsets pipeline (Job) to S3 buckets ?

Currently logs are only available at log console of Streamsets UI only

Answer 1

Looking at the code streamsets uses log4j for all its logging, so you could use something like https://github.com/parvanov/log4j-s3 and create a log write appender.

an alternative is that you could create a new pipeline to consume the logs and write to s3.

Answer 2

Logs are stored in $SDC_HOME/log directory on the VM running SDC. from there you can copy them to S3. Or you can configure your own path, and map it to an S3 bucket on OS level.