Highland consume and toArray method together

Question

Hi people and happy holidays!

I'm trying to consume a stream of csv rows with highland. To do some special processing and avoid passing down the headers to the stream, I'm calling .consume() and then I wanted the outcome in an array. The problem is that the callback of .toArray() is never called. I'm just curious about this as I already changed my solution to .map().toArray() but I still think .consume() would be a more elegant solution. ;) This is the piece of code that reads from a csv file and processes the rows:

const fs = require('fs');
const _ = require('highland');

const dataStream = fs.createReadStream('file.csv', 'utf8');

_(dataStream)
  .split()
  .consume((err, row, push, next) => {
    console.log('consuming a row', row); // <-- this shows data

    if (err) {
      push(err);
      return next();
    }

    // This condition is needed as .consume() passes an extra {} at the end of the stream <-- donno what this happens!!
    if (typeof row !== 'string') {
      return next();
    }

    const cells = row.split(',');

    if (cells[0] === 'CODE') { // <-- header row
      const { columnNames, columnIndexes } = processHeader(cells)); // <-- not relevant, works okay
      return next();
    }

    console.log('processin content here', processRow(cells)); // <-- not relevant, works okay
    push(null, processRow(cells));
    return next();
  })
  .toArray(rows => console.log('ROWS: ', rows)); // <-- I don't get anything here

Thanks a lot!

Answer 1

Both consume and toArray consume a stream. You can only consume a stream once. If you try to consume it multiple times the "stream" will be empty.