Supersonic query engine usage in a valconic query model

[Hanumanth Maduri]

Hello,

I am trying to use supersonic query engine to do database processing.

I could see examples of adding all the rows and then execute the tree of operations.

What I want to do is instead of adding all the rows and running the operations, I am looking for the supersonic query plan request the rows one by one.

Can I do that with supersonic engine.

If we add the rows all at once then it is possible that memory bloats up for large number of rows.

Thanks,

[Onufry Wojtaszczyk]

Requesting rows one by one is a bad idea - Supersonic is optimized for processing rows in "blocks" of larger size (typically around 1K rows). All methods to pull data from a Cursor return data block-by-block.

Most Supersonic input cursors (that is, cursors through which data enters the processing engine) also read in block-by-block on demand. Depending on the format you have your data in you should look whether the appropriate cursor implementation behaves this way; if your data format is unsupported you will have to write your own input cursor (it's typically easy). We encourage you to write this in a block-by-block basis, and to contribute your cursor to Supersonic.

Onufry

--
You received this message because you are subscribed to the Google Groups "Supersonic Query Engine Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to supersonic-query-...@googlegroups.com.
To post to this group, send an email to supersonic-...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[ptab]

Hi Hanumanth, 

Of course you can have your custom data source. The idea is that you need to provide your custom Cursor implementation: 

https://code.google.com/p/supersonic/source/browse/supersonic/cursor/base/cursor.h

Probably by inheriting from BaseCursor interface: 

https://code.google.com/p/supersonic/source/browse/supersonic/cursor/infrastructure/basic_cursor.h

The only thing that you need to implement is the "Next" method: 

virtual ResultView Next(rowcount_t max_row_count) = 0;

And this is exactly what you need. This way other supersonic operations, cursors will be able to ask your custom cursor for "max_row_count" rows. 

When you have this cursor implemented, you might implement your custom Operation:

https://code.google.com/p/supersonic/source/browse/supersonic/cursor/infrastructure/basic_operation.h#49

that will be just factory class for your custom cursor. 

This way you have your own operation that can use as any other in supersonic query plan.

Piotr