Stan IO / CmdStan / Protocol Buffers
Krzysztof Sakrejda
1) Has anyone tried to write up Protocol Buffers .proto files for Stan's _input_ arguments?
2) Is the argument hierarchy defined in the CmdStan manual the most authoritative thing to go from for writing the .proto files?
Background:
I (finally) had some time to figure out how CmdStan's command.hpp works and I think I can see how to do a branch that does streaming I/O through Protocol Buffers. I'm thinking of this as doing a mock-up of the I/O issues that would come up in doing a server version of Stan. My steps would be:
1) Define .proto files for CmdStan's command-line argument tree
2) Replace the argument parser (basically a temporary hack) with something to build up protocol buffers messages from command line arguments.
3) Replace the parser-argument logic in command.hpp (stuff like if (parser.arg("method")->arg("sample")) { ) with logic based on arguments extracted from the protocol buffers messages.
4) Define .proto files for CmdStan's output, I'm thinking output messages would be something like original input messages + adaptation parameters + sample/optimizer/etc... messages.
5) Modify the command.hpp output to produce protobuf messages and see about getting R to swallow them (R just because I know how to write interface code for R and there's a protocol buffers library)...
I think that path would let me test things well enough at each step so that I don't run into problems...
Krzysztof
Bob Carpenter
we need to be careful to target what it will look like, not
just what it's like now.
1) Not that I know of, but we've been thinking about it.
Allen's thoughts were that if we could define a protocol buffer
for input and output, it'd be easy to write a container for Stan
and call it that way.
2) Code is always more authoritative than doc. You can run it
and it's self documenting --- that should be most up to date
because it's mirroring the code.
- Bob
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
Michael Betancourt
The arguments will be the last change but I see no reason to use protocol buffers
for them — it’ll be possible to wrap the new arguments in something that converts
from a protocol buffer, but having to unpack a buffer for every interface will be an
unnecessary burden. Using protocol buffers for input (for example data) is another
story, but that will have to wait until at least the var_context refactor. I went through
Google’s protocol buffer implementation a few weeks ago and it would be
straightforward to write an interface_callbacks::base_writer implementation that
writes to a specified buffer.
Krzysztof Sakrejda
> Warning: The command branch is being refactored, so
> we need to be careful to target what it will look like, not
> just what it's like now.
Understood. I've been following the refactor and I think I know what's going on. Having the writer refactor go through should make things a lot simpler but I don't see it being time consuming to deal with changes.
> 1) Not that I know of, but we've been thinking about it.
> Allen's thoughts were that if we could define a protocol buffer
> for input and output, it'd be easy to write a container for Stan
> and call it that way.
Ok, good to know. I'll give it a shot and see what people say.
> 2) Code is always more authoritative than doc. You can run it
> and it's self documenting --- that should be most up to date
> because it's mirroring the code.
That's why I was thinking the right step after defining the .proto files would be to splice message construction/use into the middle of CmdStan so the two could be compared. I know there isn't much end-to-end testing but this might be the
>
> - Bob
Krzysztof Sakrejda
> As Bob notes, this is all changing (more slowly than would be ideal, but that’s life).
> The arguments will be the last change but I see no reason to use protocol buffers
> for them — it’ll be possible to wrap the new arguments in something that converts
> from a protocol buffer, but having to unpack a buffer for every interface will be an
> unnecessary burden.
>Using protocol buffers for input (for example data) is another
> story, but that will have to wait until at least the var_context refactor.
std::fstream data_stream(data_file.c_str(),
std::fstream::in);
stan::io::dump data_var_context(data_stream);
So there's a straightforward protobuf equivalent through the auto-generated C++ interface with this signature: inline const ::std::string& data() const;
> I went through
> Google’s protocol buffer implementation a few weeks ago and it would be
> straightforward to write an interface_callbacks::base_writer implementation that
> writes to a specified buffer.
Krzysztof
Allen B. Riddell
data back and forth from the interfaces to the Stan C++ routines. But if
one can write a tiny shim for this, that's fine by me. I agree with Bob
that we need to target the final state of the command branch.
Maybe it would make sense to talk about this during one of the weekly
meetings? Or perhaps the refactor special interest group needs its own
temporary mailing list or some kind of meta task on Github?
https://github.com/stan-dev/stan/pull/1546 is a little overwhelming.
Would it be possible to break that into subtasks or create a meta issue
where we can see what has been finished and what is still in progress?
Thanks,
Allen
Allen B. Riddell
suggestion at this point would be to use Protocol Buffers v3 (not v2)
since v3 introduces a JSON serialization format.
On 10/03, Krzysztof Sakrejda wrote:
Krzysztof Sakrejda
> I'd be happy to look over anything you work up Krzysztof. My only
> suggestion at this point would be to use Protocol Buffers v3 (not v2)
> since v3 introduces a JSON serialization format.
Thanks, I'll put what I can together and drop a link here. I had some surprises land in my lap recently so it might be next week.