How to define an optional field in protobuf 3

Question

I need to specify a message with an optional field in protobuf (proto3 syntax). In terms of proto 2 syntax, the message I want to express is something like:

message Foo {
    required int32 bar = 1;
    optional int32 baz = 2;
}

From my understanding "optional" concept has been removed from syntax proto 3 (along with required concept). Though it is not clear the alternative - using the default value to state that a field has not been specified from the sender, leaves an ambiguity if the default value belongs to the valid values domain (consider for example a boolean type).

So, how am I supposed to encode the message above? Thank you.

Answer 1

Since protobuf release 3.15, proto3 supports using the optional keyword (just as in proto2) to give a scalar field presence information.

syntax = "proto3";

message Foo {
    int32 bar = 1;
    optional int32 baz = 2;
}

A has_baz()/hasBaz() method is generated for the optional field above, just as it was in proto2.

Under the hood, protoc effectively treats an optional field as if it were declared using a oneof wrapper, as CyberSnoopy’s answer suggested:

message Foo {
    int32 bar = 1;
    oneof optional_baz {
        int32 baz = 2;
    }
}

If you’ve already used that approach, you can now simplify your message declarations (switch from oneof to optional) and code, since the wire format is the same.

The nitty-gritty details about field presence and optional in proto3 can be found in the Application note: Field presence doc.

Historical note: Experimental support for optional in proto3 was first announced on Apr 23, 2020 in this comment. Using it required passing protoc the --experimental_allow_proto3_optional flag in releases 3.12-3.14.

Answer 2

My protoc --version was out dated, I updated it manually by downloading the zip releases, and that fixed the issue.

Answer 3

Another way to encode the message you intend is to add another field to track "set" fields:

syntax="proto3";

package qtprotobuf.examples;

message SparseMessage {
    repeated uint32 fieldsUsed = 1;
    bool   attendedParty = 2;
    uint32 numberOfKids  = 3;
    string nickName      = 4;
}

message ExplicitMessage {
    enum PARTY_STATUS {ATTENDED=0; DIDNT_ATTEND=1; DIDNT_ASK=2;};
    PARTY_STATUS attendedParty = 1;
    bool   indicatedKids = 2;
    uint32 numberOfKids  = 3;
    enum NO_NICK_STATUS {HAS_NO_NICKNAME=0; WOULD_NOT_ADMIT_TO_HAVING_HAD_NICKNAME=1;};
    NO_NICK_STATUS noNickStatus = 4;
    string nickName      = 5;
}

This is especially appropriate if there is a large number of fields and only a small number of them have been assigned.

In python, usage would look like this:

import field_enum_example_pb2
m = field_enum_example_pb2.SparseMessage()
m.attendedParty = True
m.fieldsUsed.append(field_enum_example_pb2.SparseMessages.ATTENDEDPARTY_FIELD_NUMBER)

Answer 4

you can find if one has been initialized by comparing the references with the default instance:

GRPCContainer container = myGrpcResponseBean.getContainer();
if (container.getDefaultInstanceForType() != container) {
...
}

Answer 5

One way is to optional like described in the accepted answer: https://stackoverflow.com/a/62566052/1803821

Another one is to use wrapper objects. You don't need to write them yourself as google already provides them:

At the top of your .proto file add this import:

import "google/protobuf/wrappers.proto";

Now you can use special wrappers for every simple type:

DoubleValue
FloatValue
Int64Value
UInt64Value
Int32Value
UInt32Value
BoolValue
StringValue
BytesValue

So to answer the original question a usage of such a wrapper could be like this:

message Foo {
    int32 bar = 1;
    google.protobuf.Int32Value baz = 2;
}

Now for example in Java I can do stuff like:

if(foo.hasBaz()) { ... }

Answer 6

In proto3, all fields are "optional" (in that it is not an error if the sender fails to set them). But, fields are no longer "nullable", in that there's no way to tell the difference between a field being explicitly set to its default value vs. not having been set at all.

If you need a "null" state (and there is no out-of-range value that you can use for this) then you will instead need to encode this as a separate field. For instance, you could do:

message Foo {
  bool has_baz = 1;  // always set this to "true" when using baz
  int32 baz = 2;
}

Alternatively, you could use oneof:

message Foo {
  oneof baz {
    bool baz_null = 1;  // always set this to "true" when null
    int32 baz_value = 2;
  }
}

The oneof version is more explicit and more efficient on the wire but requires understanding how oneof values work.

Finally, another perfectly reasonable option is to stick with proto2. Proto2 is not deprecated, and in fact many projects (including inside Google) very much depend on proto2 features which are removed in proto3, hence they will likely never switch. So, it's safe to keep using it for the foreseeable future.

Answer 7

Just use:

syntax = "proto3";

message Hello {
    int64 required_id = 1;
    optional int64 optional_id = 2;
}

In Go it builds struct with

type Hello struct {
   ...
   RequiredId int64 ...
   OptionalId *int64 ...
   ...
}

You can easily check for nil and distinguish between default value (zero) and unset value (nil).

Most answers here are obsolete and unnecessarily complicated.

Answer 8

Based on Kenton's answer, a simpler yet working solution looks like:

message Foo {
    oneof optional_baz { // "optional_" prefix here just serves as an indicator, not keyword in proto2
        int32 baz = 1;
    }
}

Answer 9

To expand on @cybersnoopy 's suggestion here

if you had a .proto file with a message like so:

message Request {
    oneof option {
        int64 option_value = 1;
    }
}

You can make use of the case options provided (java generated code):

So we can now write some code as follows:

Request.OptionCase optionCase = request.getOptionCase();
OptionCase optionNotSet = OPTION_NOT_SET;

if (optionNotSet.equals(optionCase)){
    // value not set
} else {
    // value set
}

Answer 10

Another way is that you can use bitmask for each optional field. and set those bits if values are set and reset those bits which values are not set

enum bitsV {
    baz_present = 1; // 0x01
    baz1_present = 2; // 0x02

}
message Foo {
    uint32 bitMask;
    required int32 bar = 1;
    optional int32 baz = 2;
    optional int32 baz1 = 3;
}

On parsing check for value of bitMask.

if (bitMask & baz_present)
    baz is present

if (bitMask & baz1_present)
    baz1 is present