Problem in reading event stream in AWS Lambda. Nodejs code working locally as desired but not in AWS Lambda

Question

Here's the workflow:

Get a https link --> write to filesystem --> read from filesystem --> Get the sha256 hash.

It works all good on my local machine running node 10.15.3 But when i initiate a lambda function on AWS, the output is null. Some problem may lie with the readable stream. Here's the code. You can run it directly on your local machine. It will output a sha256 hash as required. If you wish to run on AWS Lambda, Comment/Uncomment as marked.

//Reference: https://stackoverflow.com/questions/11944932/how-to-download-a-file-with-node-js-without-using-third-party-libraries



var https = require('https');
var fs = require('fs');
var crypto = require('crypto')
const url = "https://upload.wikimedia.org/wikipedia/commons/a/a8/TEIDE.JPG"
const dest = "/tmp/doc";
let hexData;


async function writeit(){
  var file = fs.createWriteStream(dest);
  return new Promise((resolve, reject) => {
    var responseSent = false;
    https.get(url, response => {
      response.pipe(file);
      file.on('finish', () =>{
        file.close(() => {
          if(responseSent)  return;
          responseSent = true;
          resolve();
        });
    });

}).on('error', err => {
        if(responseSent)  return;
        responseSent = true;
        reject(err);
    });
  });

}



const readit = async () => {

await writeit();

var readandhex = fs.createReadStream(dest).pipe(crypto.createHash('sha256').setEncoding('hex'))
try {
  readandhex.on('finish', function () {       //MAY BE PROBLEM IS HERE.
    console.log(this.read())
    fs.unlink(dest, () => {});
   })
}
catch (err) {
    console.log(err);
    return err;
}
}



const handler = async() =>{                  //Comment this line to run the code on AWS Lambda
//exports.handler = async (event) => {       //UNComment this line to run the code on AWS Lambda
    try {
        hexData = readit();
    }
    catch (err) {
        console.log(err);
        return err;
    }
    return hexData;
};



handler()                                   //Comment this line to run the code on AWS Lambda

Answer 1

There can be multiple things that you need check.

1. Since, the URL you are accessing is a public one, make sure either your lambda is outside VPC or your VPC has NAT Gateway attached with internet access.

2. /tmp is valid temp directory for lambda, but you may need to create doc folder inside /tmp before using it.

You can check cloud-watch logs for more information on what's going if enabled.

Answer 2

I've seen this difference in behaviour between local and lambda before.

All async functions return promises. Async functions must be awaited. Calling an async function without awaiting it means execution continues to the next line(s), and potentially out of the calling function.

So your code:

exports.handler = async (event) => {
    try {
        hexData = readit();
    }
    catch (err) {
        console.log(err);
        return err;
    }
    return hexData;
};

readit() is defined as const readit = async () => { ... }. But your handler does not await it. Therefore hexData = readit(); assigns an unresolved promise to hexData, returns it, and the handler exits and the Lambda "completes" without the code of readit() having been executed.

The simple fix then is to await the async function: hexData = await readit();. The reason why it works locally in node is because the node process will wait for promises to resolve before exiting, even though the handler function has already returned. But since Lambda "returns" as soon as the handler returns, unresolved promises remain unresolved. (As an aside, there is no need for the writeit function to be marked async, because it doesn't await anything, and already returns a promise.)

That being said, I don't know promises well, and I barely know anything about events. So there are others things which raise warning flags for me but I'm not sure about them, maybe they're perfectly fine, but I'll raise it here just in case:

file.on('finish' and readandhex.on('finish'. These are both events, and I believe are non-blocking, so why would the handler and therefore lambda wait around for them?

In the first case, it's within a promise and resolve() is called from within the event function, so that may be fine (as I said, I don't know much about these 2 subjects so am not sure) - the important thing is that the code must block at that point until the promise is resolved. If the code can continue execution (i.e. return from writeit()) until the finish event is raised, then it won't work.

The second case is almost certainly going to be a problem because it's just saying that if x event is raised, then do y. There's no promise being awaited, so nothing to block the code, so it will happily continue to the end of the readit() function and then the handler and lambda. Again this is based on the assumption that events are non blocking (in the sense of, a declaration that you want to execute some code on some event, does not wait at that point for that event to be raised).