Integration with a protoc plugin providing extra protobuf fields validation

[Brice Jaglin]

Hi,

protoc-gen-validate, a project aiming at defining a set of rich constraints on protobuf fields and generating language-specific validators via a protoc plugin, is about to finally become polyglot and get Java support. As this feature is very promising for us to avoid having an extra, non-collocated layer of stateless business validation in our protobuf-declared, ScalaPB-served API, we started thinking about how far a ScalaPB integration could go, and potential implementations. I wanted to present three alternatives to get feedback about the implementation feasibility and cost before potentially diving in.

In any case (except 2.1), the protoc-gen-validate binaries would need to be packaged in the same way as protoc.

1. Validation could be invoked on messages converted to the native Java format. Provided everything is available to customize the call to protoc, this would not require any direct coupling between ScalaPB and protoc-gen-validate, but would incur a lot of runtime overhead (which for us, is not a big issue at the moment since incoming payloads are rather small).
   
2. Validators could be re-implemented against ScalaPB generated classes:
1. either in a separate ScalaPB module/lib (exposed via a ScalaPB flag?),
2. either as a new template/language in protoc-gen-validate's repo, which seems possible and desirable for reducing boilerplate (by reusing the static Java validators), but more complicated as it would require ScalaPBC to be used instead of protoc there for the scala_out command, effectively creating a cyclic dependency between the projects.
   

Any thoughts? Thanks!

Brice

[Nadav Samet]

Hi Brice,

I think that it would be easiest to follow (2)(1) - it's pretty easy to add a new code generator that has access to ScalaPB data model (through DescriptorImplicits.scala). See an example here: https://github.com/thesamet/sbt-protoc/blob/master/examples/custom-gen/project/MyGenerator.scala. The code generator should depend on ScalaPB and on PGV's validate.proto.

The generated code can either depend on the new PGV's Java validator at runtime, or this could be reimplemented in Scala. The advantage of the latter is that it would support for Scala.js and Scala Native, but that can come later.

The challenge with (2)(2) is that you'd have to somehow get access to the data provided by DescriptorImplicits.scala within PGV (in order to have the Scala names of things), it looks like that would have to get reimplemented in Go which I think would be brittle to match all the nuances and special cases (how naming conflicts get resolved).

Thanks for working on this. Feel free to ask if you have any further questions. And let me know once this is ready, we can put a link to it from the ScalaPB docs.

-Nadav

--
You received this message because you are subscribed to the Google Groups "ScalaPB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scalapb+u...@googlegroups.com.
To post to this group, send email to sca...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scalapb/0c122c45-a662-4788-bc80-f92d4373cb76%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

-Nadav

[Nadav Samet]

BTW, I put together a g8 template that can help you get started writing your own code generator easier. It is based on the custom-gen project linked above: https://github.com/scalapb/scalapb-plugin-template.g8

--

-Nadav

[Brice Jaglin]

Great, thanks for the pointers and the g8 template. https://github.com/lyft/protoc-gen-validate/pull/108 is shaping up quite well, so I will give a try at implementing something against it in the coming weeks.

Brice

[Brice Jaglin]

Following-up on this a year later for visiiblity: we finally got back to this, and ended up injecting PGV support into ScalaPB generated code via protobuf-java code/validators (1), leveraging support for maven code generators in sbt-protoc and adding some adhoc heuristics to skip compilation of "unnecessary" Java classes (as Java conversions add a significant overhead in compile time and bytecode size).

We have taken the controversial way of enforcing validation right in ScalaPB generated code constructors by defining our own Generator inheriting (and inlining parts of) ScalaPB's canonical ProtobufGenerator. If/when we see traction in usage of PGV internally, we might explore (2)(1) as suggested above, as well as open a ticket to discuss if/how validation hooks in ScalaPB generated code could be made easier, as the current solution is clearly a hack that I cannot easily open-source, depending on internal APIs, and fragile to any version bump.

Brice