c++ ShortDebugString()

[agallego]

Coming form the protobuf world, it's really nice to have the shortdebugstring() method. 

Now that we have the flexbuffer (reflection) impl is there a way to turn a flatbuffer -> json string that you can print or anything to dumpt he contents of a flatbuffer to the console.

I just looked at my code and I have a couple of 

namespace std {
ostream &operator<<(ostream &o, const ::smf::wal::tx_get_request &r);
ostream &operator<<(ostream &o, const ::smf::wal::tx_get_reply &r);
ostream &operator<<(ostream &o, const ::smf::wal::tx_put_request &r);
ostream &operator<<(ostream &o, const ::smf::wal::tx_put_reply &r);


ostream &operator<<(ostream &o, const ::smf::wal::wal_header &h);
ostream &operator<<(ostream &o, const ::smf::wal_read_request &r);
ostream &operator<<(ostream &o, const ::smf::wal_read_reply &r);
ostream &operator<<(ostream &o, const ::smf::wal_write_request &r);
ostream &operator<<(ostream &o, const ::smf::wal_write_reply &r);
}

type things in my source code.

I did look through the source and found no obvious way of taking a generic flatbuffer's generated table and printing the contents to std::cout for example. 

any pointers appreciated.

- Alex

[Wouter van Oortmerssen]

The current way of doing so is by calling GenerateText(). That however requires a Parser, i.e. you must parse the schema before, which may be inconvenient. If you look at registry.h, there's a nice helper class for this.

Alternatively, we could use a bit of generated code to allow us to print a FlatBuffer without loading any schemas. I happen to be working on exactly that at the moment, and it should be out on github shortly.

--
You received this message because you are subscribed to the Google Groups "FlatBuffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flatbuffers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alexander Gallego]

Sent from my mobile, please excuse my handwriting.

On Aug 28, 2017 11:39 AM, "Wouter van Oortmerssen" <w...@google.com> wrote:

The current way of doing so is by calling GenerateText(). That however requires a Parser, i.e. you must parse the schema before, which may be inconvenient. If you look at registry.h, there's a nice helper class for this.

Alternatively, we could use a bit of generated code to allow us to print a FlatBuffer without loading any schemas. I happen to be working on exactly that at the moment, and it should be out on github shortly.

Yes! Amazing. 

For binary fields it would be super helpful to display the size instead of raw contents. 

2 cents in case you were looking for community feedback. 

Looking forward to it. 

Loading the schema, is a no go since the runtime won't have access to those schemas in my case. 

Thanks!!! 

[Wouter van Oortmerssen]

By default it will try to print everything looking as closely as the current JSON output, including byte arrays. But I suppose we could have an option to not print byte arrays and/or have have a cut-off value to not print any arrays over a certain size.

The good news is that the way I am implementing it allows anyone to write custom printing / flatbuffer iteration code, so you could adapt it to your needs (the generated code only contains a "mini reflection table" for each type).

[Alexander Gallego]

Got it. Will have to see how to adapt it then. I guess a simple visitor lambda per field would allow custom printing. 

I.e.: 

auto p = [ ]( auto x ) { return x; }

auto bprint = [ ] (vector<uint8t>& x){ return x.size(); }

And just let the runtime type dispatch correctly. 

Cool!!

Thanks again. 

Sent from my mobile, please excuse my handwriting.

[Maxim Zaks]

A crazy idea I had but did not find time to implement it yet is to generate a dot file (https://de.wikipedia.org/wiki/DOT_(GraphViz))

This could be a great diagnostic tool, showing vTables, color coding validation errors. A node id is the absolute offset in the buffer this way it can graphically represent DAGs.

For sure it could represent only buffer of certain size, I think it would be feasible to generate multiple diagrams files based on number of nodes.

So something like diagramms I used in my talks to explain the internals of FlatBuffers (https://cdn-images-1.medium.com/max/800/1*G4tyetn9Nhd5rV-BKcRIKg.png)

Well with a little bit more information like the Table name, struct name field names etc...

To unsubscribe from this group and stop receiving emails from it, send an email to flatbuffers...@googlegroups.com.

[Alexander Gallego]

On Thu, Aug 31, 2017 at 8:43 AM, 'Maxim Zaks' via FlatBuffers <flatb...@googlegroups.com> wrote:

A crazy idea I had but did not find time to implement it yet is to generate a dot file (https://de.wikipedia.org/wiki/DOT_(GraphViz))

This could be a great diagnostic tool, showing vTables, color coding validation errors. A node id is the absolute offset in the buffer this way it can graphically represent DAGs.

For sure it could represent only buffer of certain size, I think it would be feasible to generate multiple diagrams files based on number of nodes.

Ha that's very cool - i built something like that for a stream processor (concord.io) in a past life for tracking DAG flows. 

But most of the time, i'd say i want something on my console i can put under a debug log. 

 

So something like diagramms I used in my talks to explain the internals of FlatBuffers (https://cdn-images-1.medium.com/max/800/1*G4tyetn9Nhd5rV-BKcRIKg.png)

Well with a little bit more information like the Table name, struct name field names etc...

It would be nice to see the output of your nesting of tables - i.e.: allow yout detect performance tradeoffs and see if you need to flatten your table hierarchy. 

Neat! 

 

--
You received this message because you are subscribed to a topic in the Google Groups "FlatBuffers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/flatbuffers/nCutIdI3a4U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to flatbuffers+unsubscribe@googlegroups.com.

[agallego]

hi wouter, 

any progress on the pretty print, only mentioned it because you said you were working on it. 

Thanks! 

[Wouter van Oortmerssen]

any progress on the pretty print, only mentioned it because you said you were working on it. 

Some delays, but I am indeed very close to done.. just have to make sure it prints enums correctly. Likely done by the end of this week, or otherwise next week.

[Wouter van Oortmerssen]

It just landed here, let me know what you think: https://github.com/google/flatbuffers/commit/72a99abfb7db64dc49720b28b41f382b5ec7cde0

[Alexander Gallego]

Looks great !!!

though I use the object api as much as I use the regular API and for the object API (from a cursory read) you'd have to 

```

flatbuffers::builder fbb

fbb.Finish( RootType::Pack( fbb, objectApiPtr) )

flatbuffers::FlatbuffersToString( fbb.GetBufferPointer,  MyTypeTable() )

``` 

which would be too costly, just for printing  - 1 the print buffer allocation + the unnecessary packing of the object API) allocations

I think there needs to be other code-gen for the object API as well - which need not match in output necessarily. 

maybe i missed the object api printing too 

Looking fresh! 

[Wouter van Oortmerssen]

Yeah, there's no printing for the Object API, and there's no way to adopt any of the current reflection methods to it (since you can't programmatically access an Object API field).

To do printing for the Object API, you'd have to generate yet more code. But rather than hard-coding it to printing, it should probably use a visitor similar to the current mini-reflection code, such that it can be used for other purposes too.

[agallego]

Got it! 

It would be super useful to have the object api ShortDebugString() stuff as well, but even for the regular API is great! 

Thanks

[agallego]

I didn't see docs for implementing the 'visitor' or example code in the test.cpp 

Currently this is great!  - but i want to change the array printout to something else - mostly the size instead of the contents. 

did i miss a test for this ? INFO  2017-09-27 10:26:09,047 [shard 0] smf - chain_replication_service.cc:62] Chain put req: { chain: [ 2130706433 ], chain_index: 1, put: { topic: "dummy_topic", data: [ { partition: 520, txs: [ { op: full, type: kv, epoch_ms: 246606866, kv: { key: [ 88, 125, 41, 113, 80, 57, 79, 122, 117, 68, 88, 73, 91, 70, 49, 78, 56, 81, 72, 41, 45, 50, 42, 36, 75, 114, 115, 64, 41, 112, 61, 106, 109, 126, 112, 52, 94, 97, 57, 125, 71, 40, 77, 43, 80, 103, 126, 105, 65, 109, 77, 38, 113, 119, 65, 85, 75, 80, 122, 57, 90, 55, 40, 84, 76, 36, 124, 33, 41, 53, 122, 51, 110, 103, 78, 70, 100, 113, 52, 33, 113, 49, 111, 71, 87, 66, 74, 76, 119, 58, 61, 53, 60, 65, 46, 39, 45, 123, 93, 66, 112, 67, 95, 115, 108, 70, 102, 56, 122, 77, 96, 121, 121, 120, 46, 75, 126, 61, 65, 38, 125, 95, 60, 76, 104, 70, 93, 48, 68, 84, 66, 89, 98, 37, 95, 125, 106, 52, 101, 37, 51, 39, 43, 117, 76, 46, 110, 88, 72, 74, 72, 59, 123, 101, 50, 68, 121, 61, 102, 96, 67, 122, 113, 100, 97, 60, 116, 116, 55, 62, 41, 96, 43, 50, 106, 38, 118, 80, 46, 49, 62, 90, 34, 107, 47, 42, 50, 116 ], value: [ 97, 114, 38, 35, 86, 40, 85, 117, 100, 100, 118, 33, 124, 87, 93, 124, 83, 58, 42, 108, 38, 72, 125, 57, 102, 65, 54, 72, 51, 95, 76, 46, 45, 100, 43, 69, 103, 56, 76, 79, 97, 87, 88, 106, 114, 47, 102, 55, 103, 70, 57, 45, 114, 123, 114, 60, 49, 52, 62 ] } } ] } ] } }

INFO  2017-09-27 10:26:09,047 [shard 0] smf - chain_replication_service.cc:62] Chain put req: { chain: [ 2130706433 ], chain_index: 1, put: { topic: "dummy_topic", data: [ { partition: 520, txs: [ { op: full, type: kv, epoch_ms: 246606866, kv: { key: [ 88, 125, 41, 113, 80, 57, 79, 122, 117, 68, 88, 73, 91, 70, 49, 78, 56, 81, 72, 41, 45, 50, 42, 36, 75, 114, 115, 64, 41, 112, 61, 106, 109, 126, 112, 52, 94, 97, 57, 125, 71, 40, 77, 43, 80, 103, 126, 105, 65, 109, 77, 38, 113, 119, 65, 85, 75, 80, 122, 57, 90, 55, 40, 84, 76, 36, 124, 33, 41, 53, 122, 51, 110, 103, 78, 70, 100, 113, 52, 33, 113, 49, 111, 71, 87, 66, 74, 76, 119, 58, 61, 53, 60, 65, 46, 39, 45, 123, 93, 66, 112, 67, 95, 115, 108, 70, 102, 56, 122, 77, 96, 121, 121, 120, 46, 75, 126, 61, 65, 38, 125, 95, 60, 76, 104, 70, 93, 48, 68, 84, 66, 89, 98, 37, 95, 125, 106, 52, 101, 37, 51, 39, 43, 117, 76, 46, 110, 88, 72, 74, 72, 59, 123, 101, 50, 68, 121, 61, 102, 96, 67, 122, 113, 100, 97, 60, 116, 116, 55, 62, 41, 96, 43, 50, 106, 38, 118, 80, 46, 49, 62, 90, 34, 107, 47, 42, 50, 116 ], value: [ 97, 114, 38, 35, 86, 40, 85, 117, 100, 100, 118, 33, 124, 87, 93, 124, 83, 58, 42, 108, 38, 72, 125, 57, 102, 65, 54, 72, 51, 95, 76, 46, 45, 100, 43, 69, 103, 56, 76, 79, 97, 87, 88, 106, 114, 47, 102, 55, 103, 70, 57, 45, 114, 123, 114, 60, 49, 52, 62 ] } } ] } ] } }

[agallego]

ah nvm. 

one can extend the ToStringVisitor 

inline std::string FlatBufferToString(const uint8_t *buffer,
	const TypeTable *type_table) {
ToStringVisitor tostring_visitor;
IterateFlatBuffer(buffer, type_table, &tostring_visitor);
return tostring_visitor.s;
}