Error: Failure: Invalid wire type: %s (at position %s) when deserializing protobuf data stored in redis 7 using latest npm redis package versioned 4.6

Question

I had been using redis npm package versioned 3.1.2 in my Node.js application. Now i am trying to upgrade redis npm package to 4.6.13 and since this is a major upgrade, things that used to work with version 3.1.2 has broken with 4.6.13. New redis server installation has also been upgraded to latest available version of 7

The problem mainly arises when i am trying to deserialize the protobuf data stored in redis.

I have created a sample code:

Example code that i am using is:

set proto message:

 
 const setApplicationInfo = (data) => {
     try {
        const applicationDetailMessage = new ApplicationDetails();
        applicationDetailMessage.setEmail(data.email);
        applicationDetailMessage.setHomephonenumber('');
        applicationDetailMessage.setMobilephonenumber(data.primaryPhoneNumber);
        applicationDetailMessage.setYearofbirth(data.yearOfBirth);
        applicationDetailMessage.setFirstname(data.firstName);
        applicationDetailMessage.setMiddlename(data.middleName);
        applicationDetailMessage.setLastname(data.lastName);
        applicationDetailMessage.setMonthofbirth(data.monthOfBirth);
        applicationDetailMessage.setDayofbirth(data.dayOfBirth);
        applicationDetailMessage.setSsn(data.ssn);
        applicationDetailMessage.setStreetaddress(data.streetAddress1);
        applicationDetailMessage.setCity(data.city);
        applicationDetailMessage.setState(data.stateCode);
        applicationDetailMessage.setZipcode(data.zipCode);
        applicationDetailMessage.setResidencetype(data.residenceType);
        // applicationDetailMessage.setResidencelengthinyears(data.durationOfResidenceYear);
        // applicationDetailMessage.setResidencelengthinmonths(data.durationOfResidenceMonth);
        const serializedData = applicationDetailMessage.serializeBinary();
        return Buffer.from(serializedData);
     } catch (err) {
         throw new Error(err)
     }
 };

get redis connection:

 const getRedisConnection  = async () => {
     try {
          const client = await createClient({
               url: `redis://${process.env.REDIS_USER}:${process.env.REDIS_PASS}@${process.env.REDIS_HOST}:${process.env.REDIS_PORT}`,
            //    legacyMode: true
          })
          client.on('error', err => console.log('Redis Client Error', err))
          await client.connect();
          await client.ping();
          return client;
     } catch (err) {
          throw err;
     }
}

main function doing serializing and deserializing the proto stuffs

(async () => {
    try {
        const redisConn = await getRedisConnection();
        const id  = '1';
        const data = {
            email: '[email protected]',
            primaryPhoneNumber: '4078790329',
            dayOfBirth: 11,
            monthOfBirth: 12,
            yearOfBirth: 1996,
            firstName: 'Vernon',
            middleName: '',
            lastName: 'McClelland',
            ssn: '228412571',
            streetAddress1: '4311 Grand Avenue',
            city: 'Orlando',
            stateCode: 'FL',
            zipCode: '32810',
            residenceType: 1,
            durationOfResidenceYear: 2,
            durationOfResidenceMonth: 1,
        }
        const message = setApplicationInfo(data)
        const deserializedApplicationObj = ApplicationDetails.deserializeBinary(new Uint8Array(message)).toObject();
        console.log('deserializedApplicationObj', deserializedApplicationObj);
        await redisConn.set(`application_detail_${id}`, message);

        const applicationRes = await redisConn.get(`application_detail_${id}`);
        const applicationBufferData = Buffer.from(applicationRes, 'utf-8');
        console.log('applicationBufferData',  applicationBufferData);
        if (applicationBufferData && applicationBufferData.length > 0) {
            const applicationObj = ApplicationDetails.deserializeBinary(new Uint8Array(applicationBufferData)).toObject();
            console.log('applicationObj', applicationObj);
        }
    } catch (err) {
        throw err;
    }
})()

This works fine and it deserializes the application object successfully

logged application obj is:

applicationObj {
  email: '[email protected]',
  homephonenumber: '',
  mobilephonenumber: '4078790329',
  yearofbirth: 32464879,
  firstname: 'Vernon',
  middlename: '',
  lastname: 'McClelland',
  monthofbirth: 12,
  dayofbirth: 11,
  ssn: '228412571',
  streetaddress: '4311 Grand Avenue',
  city: 'Orlando',
  state: 'FL',
  zipcode: '32810',
  residencetype: 1,
  residencelengthinyears: 0,
  residencelengthinmonths: 0
}

Now in the function setApplicationInfo, when i uncomment following line of code:

// applicationDetailMessage.setResidencelengthinyears(data.durationOfResidenceYear);

It throws error:

Note:

before storing the serialized proto buffer message to redis server, i was checking to see if deserialization works or not for that particular proto buffer and it works fine upto that point,

        const deserializedApplicationObj = ApplicationDetails.deserializeBinary(new Uint8Array(message)).toObject();
        console.log('deserializedApplicationObj', deserializedApplicationObj);

it only throws error when i try to deserialize the data fetched from redis server

Error message is as:

 jspb.asserts.fail=function(a,b){for(var c=[],d=1;d<arguments.length;++d)c[d-1]=arguments[d];throw Error("Failure"+(a?": "+a:""),c);};jspb.asserts.assertInstanceof=function(a,b,c,d){for(var e=[],f=3;f<arguments.length;++f)e[f-3]=arguments[f];a instanceof b||jspb.asserts.doAssertFailure("Expected instanceof %s but got %s.",[jspb.asserts.getType(b),jspb.asserts.getType(a)],c,e);return a};
                                                                                                  ^

Error: Failure: Invalid wire type: %s (at position %s)
    at jspb.asserts.fail (protos/node_modules/google-protobuf/google-protobuf.js:90:99)
    at jspb.BinaryReader.nextField (/protos/node_modules/google-protobuf/google-protobuf.js:384:536)
    at proto.demo.entities.application_pb.ApplicationDetails.deserializeBinaryFromReader (protos/build/application_pb.js:24459:17)
    at proto.demo.entities.application_pb.ApplicationDetails.deserializeBinary (protos/build/application_pb.js:24447:66)
    at /del_v2/redis/testredis.js:93:53
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

Node.js v20.9.0

What could be the reason for this error? Issues happens only with buffer data stored in redis while deserializing.

Answer 1

I think the problem is related to the following lines:

const applicationRes = await redisConn.get(`application_detail_${id}`);
const applicationBufferData = Buffer.from(applicationRes, 'utf-8');

Line one is returning a String that contains binary data. Node Redis will encode this as if it were escaped C-style string and not UTF-8.

So, if the byte is a printable ASCII character, it'll print it. If the byte is something like a newline or a carriage return, it will return \n or \r respectively. And, if the byte is something that doesn't have a common escaped representation, it'll escape them with a \x and the value in hex. So a byte containing a value of 1 will return \x01 and a byte containing 255 will return \xff.

For example, if I have the bytes 74 65 73 74 0A 01 02 03, I will get a string of test\n\x01\x02\x03 back.

Buffer.from receives this text and incorrectly decodes it as it's not proper UTF-8. You're probably getting away with this because your data happens to line up just so. In other words, a lucky coincidence. This would likely happen at other places if you changed the data. If you hadn't caught this during development, this would have resulted in a very hard to track down intermittent error. So, a lucky coincidence indeed!

What you need to do is tell Node Redis to return a Buffer directly. Then just use that Buffer. You can do this by passing in commandOptions to your call:

const applicationBufferData = await redisConn.get(
  commandOptions({ returnBuffers: true }),
  `application_detail_${id}`
);

It's a little clunky, but this will return a Buffer containing the raw bytes, which is what you want.

Full disclosure, I haven't tested this and I could be wrong. But I do believe this is what is happening and I do work for Redis so I'm reasonably well informed. But, even if I'm wrong, I hope it was at least informative!

Best of luck!