Using Protobufs with a gRPC-Gateway

Author:Daniel B.

Some of my recent work has involved using Protocol Buffers (protobufs) through version 2 of the library protobuf-es on a frontend that interacts with a gRPC backend through a gRPC-Gateway, a reverse proxy that translates HTTP/JSON requests into gRPC. Here is some of what I have learned about protobufs, protobuf-es, and how to deal with them on the frontend through a reverse proxy like gRPC-Gateway. This pertains to protobuf-es v2 specifically, which is a major change from v1.

What is Protobuf

"Protocol buffers are Google’s language-neutral, platform-neutral, extensible mechanism for serializing structured data". protobuf.dev. In frontend terms, this is like a 'type' (called a 'message') generated from the backend code that the frontend can use. For instance, Golang might use 'int32' to specify a numeric type. Though JavaScript / TypeScript does not have a type of this specific name, protobuf-es will generate a protobuf message that tells the frontend to use 'number', a type it recognizes.

Protobuf Messages are Different From "Types"

The structures protobuf specifies are called 'Messages'. Though I think of them as types, I found that trying to use them just like TypeScript types was not possible.

Trying to build an object assigned to this type resulted in this error message:

Type '{...}' is not assignable to type 'Order'.
  Property '$typeName' is missing in type '{...}' but required in type 'Message<"orderMaker.Order">'

Where is this $typeName property coming from? It is not listed in the message. Should I add it into the object I am using? Create a utility type that omits this property? It turned out that this was not necessary. Protobuf-es requires certain metadata beyond what is explicitly listed in the message. To add this metadata, protobuf-es provides a method 'create'. To achieve type safety and ensure you are using the right type, you need to use 'create' along with the associated schema for the message. Schemas specify the structure of the data. One advantage of using a schema is its neutrality - the same schema can be used in different programming languages.

Example:

const order = create(orderSchema, {
    orderId: '123',
    orderStatus: 'FULFILLED',
    ...
})

The existence of the $typeName property illustrates a key difference between TypeScript and protobuf-es. TypeScript uses structural typing - if two types share the same properties, they are considered to be the same type. Protobuf-es uses 'nominal typing': this means that even if two types share the same properties, if they are designated as different types, they will be considered different. Protobuf-es adds $typeName as a unique identifier to each individual message, ensuring each type is considered distinct within TypeScript's structural type system.

At first I did not see the advantage of using protobuf-es - a regular type is easier to use, requiring no special methods like 'create' from protobuf-es, so what is the point? One advantage of using protobufs on the frontend is getting runtime validation of your API response, but the main advantages are on the backend: compared to JSON, protobufs are smaller and can be parsed faster. This saves data storage space and bandwidth, and results in faster communication with the server.

In my case, the API responses were being converted to JSON through a gRPC-Gateway, which negates the performance benefits. Using a library like gRPC-Web or ConnectRPC allows the browser to take better advantage of the faster communication that gRPC provides.

The BigInt Problem

I was working with protobufs generated from a backend built with Golang. The ID and count fields this backend returned in its responses were typed as "int64", a 64-bit integer data type. In JavaScript, the most familiar numeric data type is "number". The problem is, Golang and JavaScript have a mismatch - the maximum value of a "number" that JavaScript can safely deal with is 9,007,199,254,740,991, whereas the maximum of Golang's int64 is 9,223,372,036,854,775,807. Because of this, protobuf-es renders the equivalent JavaScript type as "BigInt".

It became clear that it was going to be difficult to incorporate BigInt into existing logic that was predicated on the number type, because "A BigInt value cannot be used with methods in the built-in Math object and cannot be mixed with a Number value in operations; they must be coerced to the same type." Mozilla - BigInt

BigInt can be coerced into a number type, but this is not safe to do if the BigInt is larger than "number"'s maximum value - if you do this you are going to lose some accuracy.

Common practice is to convert the 'int64' datatype to 'strings' in JavaScript. For count fields, however, the problem remains of having to convert this to a number for mathematical operations, with possible precision loss if the value is too high.

On the Golang backend, if you are sure the numbers you are dealing with are not so high that you NEED to use int64, you can use a data type that protobuf-es will convert to a JavaScript number instead; i.e., uint32 or int32. A 'count' generally cannot be negative, so uint32 is the better choice. It does not include negative numbers, but offers a higher maximum value of about 4.3 billion vs int32's approximate 2.1 billion.

ID fields serve as unique identifiers for a piece of data, so you are probably not going to do math with them. For int64 ids, turning them into JavaScript strings is appropriate. If the ids are long sequences of numbers, they could be above JavaScript's maximum safe integer value, so converting them to numbers could result in precision loss - defeating the purpose of having a unique permanent ID in the first place. Converting them to strings results in no loss of precision. You can tell protobuf-es to do this with

  int64 field = 1 [jstype = JS_STRING]; // will generate `field: string`

and adding this config to the generation YAML file:

# Add to buf.gen.yaml:
managed:
  enabled: true
  override:
    - field_option: jstype
      value: JS_STRING
      field: ... # you can use this to target a specific field; otherwise, this will apply to all int64 fields.

protobuf-es manual

Note: This Requires Numerical Strings

In one instance, tests I created were not working properly. The cause was incorrectly typed data used in the tests, related to the BigInt problem.

Keep this in mind: if the original type specifies a 64-bit integer, but you use the configuration [jstype = JS_STRING] to define the type as a string in JavaScript, it needs to be a numerical string; that is, '123' and not 'abc' or '123a'. Using any nonnumerical characters will cause the protobuf-es methods to fail, and the data will not be serialized.

Why use BigInt to represent 64-bit integers?

The protobuf-es manual has this to say about it:

The short answer is that they're the best way to represent the 64-bit numerical types allowable in Protobuf. BigInt has widespread browser support, and for those environments where it isn't supported, we fall back to a string representation.

Using BigInt was a deliberate choice in order to safely represent all possible values for 64-bit numerical types, but in JavaScript I prefer to work with a string or number if possible.

• For IDs, I suggest making them the string type with [jstype = JS_STRING].
• For numerical types, if you are sure the value will not be less than -2,147,483,648 or more than 2,147,483,647, consider using int32 instead. If the values cannot be negative, uint32 is the better choice.

Issues Using gRPC-Gateways

If you are using protobuf messages through a gRPC-Gateway, the gateway serializes the protobuf response to JSON before it reaches the frontend - meaning the value the API returns will not be exactly what the Message seems to indicate.

The difficulty here is the message and its schema serve as the type's source of truth; however, the message is not exactly what the API returns, requiring the use of protobuf-es methods to convert the data.

Dealing With API calls through a gRPC-Gateway

If you are using a gRPC-Gateway and you still want to use protobuf messages as a structural source of truth for your data, protobuf-es provides methods for this.

For example, the fromJson() method. The gateway is giving you a JSON response, but you are using a message as your source of structural truth. In your fetching logic, you can convert the response from its JSON form to its message form with the protobuf-es fromJson() method and the response's corresponding schema definition. That allows you to accurately work with the data based on its message definition.

The opposite is toJson(). The shape of our request is defined as a message, but the gRPC-Gateway expects JSON. We can solve this by converting the request to JSON with toJson() and the request's associated schema definition before it is sent.

The Unexpected Enum

One particular pain point for me was dealing with enumerations defined in the Message. In one instance, neither the key nor the value of the enum was actually being returned by the API, to my consternation. This was caused by the way in which protobuf-es generates enumerations that have a shared prefix. We had an enumeration like this in our protobuf file:

export enum CatColor {
  /**
   * @generated from enum value: CAT_COLOR_CHOCOLATE = 0;
   */
  CHOCOLATE = 0;

  /**
   * @generated from enum value: CAT_COLOR_LAVENDER = 1;
   */
  LAVENDER = 1;

  /**
   * @generated from enum value: CAT_COLOR_CINNAMON = 2;
   */
  CINNAMON = 2;
}

But the actual API response was something like "CAT_COLOR_CHOCOLATE". Not the enum key "CHOCOLATE", nor the corresponding enum value 0. This caused me more trouble than it should have - the fromJson() and toJson() methods handle this by converting enums between their TypeScript enum and JSON forms - but it was a sticking point because it did not make sense to me. It turns out that the original definition of the enum was:

enum CatColor {
  CAT_COLOR_CHOCOLATE = 0;

  CAT_COLOR_LAVENDER = 1;

  CAT_COLOR_CINNAMON = 2;
}

When protobuf-es generates TypeScript code from an enum, it removes any shared prefix they might have - in this case, dropping the "CAT_COLOR" from each key. The TypeScript code did not include the prefix, but the JSON form of the response did. The protobuf file announced this origin in its comments:

@generated from enum value: CAT_COLOR_CHOCOLATE = 0;

If you do not want to worry about the protobuf-es methods, and you only want to work with what the gRPC-Gateway actually returns, you can use this setting:

"JSON_types=true"

"Generates JSON types for every Protobuf message and enumeration. Calling toJson() automatically returns the JSON type if available."

This will cause protobuf-es to include the JSON type (if available) that will actually reach the frontend.

It seems like using the JSON types alone would be more straightforward than using protobuf-es's methods to convert to and from Messages. So why might you choose to use the messages and schemas rather than the JSON types? One reason is to get runtime validation of your types.

Runtime validation

The protobuf-es method 'fromJson' is used with message schemas and will throw an error if its input does not match the schema.

One advantage of this is you can catch unexpected API changes - if for some reason the response no longer matches the schema, with proper error handling you can make it so an alert is triggered.

try {
  const orderResponse = fromJson(orderResponseSchema, {
    orderId: '123',
    orderStatus: 'FULFILLED',
    ...
  })
} catch (error) {
  showNotification("The server returned an unexpected response.")
}

You can show a descriptive error specifying what went wrong rather than failing silently or throwing a generic JavaScript TypeError.

It is useful to add error handling when protobuf methods are used - otherwise it may fail silently without communicating the cause.

Pitfall: Unexpected New Properties Cause Errors

When using fromJson(), an error will be thrown if an unexpected property is present. This may break functionality if your frontend protobufs are not continuously updated to match the backend.

Generally, if there are non-breaking changes to the backend protobufs (i.e., new properties added to the protobuf that have not yet been incorporated into the frontend), you will not want fromJson() to throw an error. Such runtime error handling is most useful to alert you to breaking changes such as the removal of or changes to the type/names of properties that are already being used on the frontend.

Fortunately, protobuf-es has an option that prevents this: ignoreUnknownFields

By default, unknown properties are rejected. This option overrides that behavior and ignores properties, as well as unrecognized enum string representations.

ignoreUnknownFields: true will prevent fromJson from throwing an error if there are new "unknown" fields present in the response.

Example:

  const orderResponse = fromJson(orderResponseSchema, {
    orderId: '123',
    orderStatus: 'FULFILLED',
    ...
  },
  { ignoreUnknownFields: true }
  )

Conclusion

Using protobufs through a gRPC-Gateway presents some challenges that can be solved through proper use of the conversion methods that protobuf-es provides. This can be useful for catching unexpected API changes at runtime. However, on the front-end the performance benefits provided by gRPC are largely negated by the gateway. If you are committed to the gateway and do not have future plans to use a library that takes advantage of gRPC's increased speed, directly using the JSON types generated by protobuf-es through the JSON_types=true option may be a simpler and more maintainable solution.