Fix #134: (protobuf) proto3 packed-default and oneof field resolution#707
Open
cowtowncoder wants to merge 4 commits into
Open
Fix #134: (protobuf) proto3 packed-default and oneof field resolution#707cowtowncoder wants to merge 4 commits into
cowtowncoder wants to merge 4 commits into
Conversation
…chemas proto3 defaults repeated scalar/enum fields to "packed" wire encoding unless explicitly overridden with packed=false; this library only ever honored an explicit packed option (defaulting to unpacked when absent), causing decode failures against real protoc-based proto3 producers. Thread the parsed schema's proto2/proto3 syntax through NativeProtobufSchema -> TypeResolver -> ProtobufField so the default "packed" value is computed correctly per spec.
during schema resolution Found while investigating #134 (proto3 compatibility): MessageElement keeps `oneof`-block fields in a separate `oneOfs()` list from regular `fields()`, and TypeResolver only ever resolved the latter, so any field inside a `oneof` group vanished from the resolved schema with no error -- it just could never be read or written. Merge `oneOfs()` member fields in alongside regular fields during resolution.
added 2 commits
July 1, 2026 21:57
…escriptor sets Found while investigating #134 (proto3 compatibility): FileDescriptorProto.getSyntax() called ProtoFile.Syntax.valueOf(syntax) directly on the raw descriptor string, but protoc writes lowercase "proto2"/"proto3" while the enum constants are PROTO_2/PROTO_3. Any proto3-origin FileDescriptorSet crashed with IllegalArgumentException instead of loading. Map the string explicitly instead of relying on valueOf().
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This started as a fix for #134 (incorrect decoding of
repeatedfields under proto3 syntax, as reported by @vogt31337), and grew to include two more related bugs found while investigating proto3 compatibility more broadly.1. Proto3 "packed by default" for repeated scalar/enum fields (#134)
proto3 defaults
repeatedscalar/enum fields to "packed" wire encoding unless explicitly overridden withpacked=false. This library only ever honored an explicitpackedoption on a field, defaulting to unpacked when none was present -- correct for proto2, but wrong for proto3, where that same absence means packed. As a result, content produced by any real protoc-based proto3 encoder (Go, Python, C++, etc.) that relies on the packed default failed to decode.The fix threads the parsed
.protofile's syntax (proto2 vs proto3) down through the schema-resolution pipeline:NativeProtobufSchema(captured fromProtoFile.syntax()) ->TypeResolver->ProtobufField, which now computes the defaultpackedvalue asrepeated && isProto3 && type.isPackable()only when no explicitpackedoption is present.Backward compatibility:
syntax = "...";statement (the common case in this repo's existing tests) getisProto3=false, i.e. unchanged behavior -- verified viaProtoFile.syntax()returningnullin that case.[packed=true]/[packed=false]options are unaffected and always take precedence, for both proto2 and proto3.FieldType.isPackable()).ProtobufField.isValidFor()already accepts both packed and unpacked wire types for a repeated field.proto3schema with no explicitpackedoption now produces packed (spec-compliant) output instead of unpacked. No existing test relied on the old proto3 write behavior.2.
oneoffields silently dropped during schema resolutionMessageElement(fromcom.squareup.protoparser) keeps fields declared inside aoneofblock in a separateoneOfs()list from regularfields().TypeResolver._resolve()only ever iteratedfields(), so any field inside aoneofgroup vanished from the resolved schema -- no error, it just could never be read or written. This affects proto2oneoftoo, not just proto3, butoneofis far more idiomatic in typical proto3 schemas.Fix: merge each
oneOf.fields()into the field list resolved byTypeResolver, alongside the message's regular fields.ProtobufField's constructor already treatedFieldElement.Label.ONE_OFcorrectly (falls through to non-required/non-repeated, i.e. optional-like) once given the field -- the bug was purely that these fields never reached that constructor at all.3.
FileDescriptorSet-based schema loading crashed on proto3 descriptor setsFound while reviewing this PR:
FileDescriptorSet.FileDescriptorProto.getSyntax()calledProtoFile.Syntax.valueOf(syntax)directly on the raw descriptor string. RealFileDescriptorProto.syntaxvalues (as written byprotoc) are lowercase ("proto2"/"proto3"), but the enum constants arePROTO_2/PROTO_3-- sovalueOf("proto3")threwIllegalArgumentExceptionfor any proto3-origin binary descriptor set, meaning theFileDescriptorSet.schemaFor()/schemaForFirstType()loading path (as opposed to parsing.prototext directly) was completely broken for proto3.Fix: map the string explicitly (
"proto3"->PROTO_3, anything else ->PROTO_2, matching the pre-existing null-handling default) instead of relying onvalueOf().Test plan
Proto3PackedDefault134Test: proto3 implicit-packed read, proto3 explicitpacked=falseoverride, proto2 default (unpacked) unaffected, no-syntax-declaration unaffected, and a proto3 write+read round trip.OneofFieldResolutionTest: verifies all fields inside aoneofblock (plus a sibling regular field) are present in the resolved schema, and round-trip correctly through write+read.FileDescriptorSetProto3Test: buildsFileDescriptorProto/DescriptorProtoobjects directly withsyntax = "proto3"/"proto2"and verifiesFileDescriptorSet.schemaFor()no longer crashes and produces the correct packed default for each.protobufmodule test suite passes with no regressions after each change.