I have node js server with a Postgres db. Im using pg-copy-streams to generate csv files and aws-sdk to upload them to S3. In other words, I have a pipeline that looks like this
const passthrough = createPassthrough();
pipeline(
hereGoesMyPgCopyStream,
passthrough,
(err) => {
if (err) {
console.log("There is an error");
} else {
console.log("pipeline successful");
}
},
);
const params = {
Bucket: bucketName,
Key: 'uploaded_file.csv',
Body: passthrough
};
s3.upload(params, (err, data) => {
if (err) {
console.error("Error uploading file:", err);
} else {
console.log("File uploaded successfully. Location:", data.Location);
}
});
It's working fine but now I need to upload not csv but xlsx files. I would like to keep using streams so I was thinking of doing something like
const passthrough = createPassthrough();
pipeline(
hereGoesMyPgCopyStream,
csvToXlxsConverterStream,
passthrough,
(err) => {
if (err) {
console.log("There is an error");
} else {
console.log("pipeline successful");
}
},
);
How can I implement the csvToXlxsConverterStream? All converting methods I've seen require the whole file. Is it even possible to do such conversion by chunks?
As I don't know much about the xlsx format I tried looking for libraries with this capability. The thing is, I'm not sure of how to do the convertion by chunks without resulting in a bad xlsx