compiler warnings, variantdatum, etc

[Botond Botyanszki]

Hello,

Thanks for releasing supersonic, this is truly a masterpiece.

While trying to use the code I ran into a few issues:

1. I like to compile my code with all compiler warnings enabled.
Unfortunately when including the supersonic headers, gcc produces a ton
of warnings. The following gcc switches need to be added to suppress
these: -Wno-shadow -Wno-conversion -Wno-unused-parameter
-Wno-switch-default -Wno-sign-compare -Wno-return-type
-Wno-ignored-qualifiers
Is there any reason you are not using these?

2. I have a use-case where some data I need to operate on is
weakly-typed, similarly to how variables in perl and some other languages
are handled. e.g. $var >= 42 would work for integers and also strings
that can be parsed as an int. The VariantDatum is mostly an ideal class
for this data, unfortunately this is unused elsewhere in the code. It
would involve quite an amount of code to introduce it into the type
system and make most expressions/operations compatible with this.
I know that this wouldn't be terribly efficient due to the overhead
involved, contrary to the design goals of supersonic, but I can't really
think of anything else right now to be able to solve this problem simpler
and less intrusively.

Another half-baked solution was to use some kind of type-casting or
parser functions which would then explicitly define the return type of
the data, e.g. int($var) >= 42
The only case this wouldn't work for example is comparing two variables,
$var1 == $var2. The expression should evaluate without an error if the two
types are the same (or can be casted/parsed), whereas doing an explicit
type-casting would fail in the following case if the vars contain values
that don't look like integers: int($var1) == int($var2)

3. In addition to the above, I need to handle some custom data types
(complex structures, like list or map) which are returned in the result
view. This is similar to case 2 above, except that there is no need to
execute any expression evaluation on this data. This could be easily
solved by adding a custom DataType to supersonic.proto and types.h, but I
thought I'd ask in case there is a solution that could work without the
need to modify the code at all.

I'm pretty sure you guys have already dealt with issues 2) and 3), there
are traces in the code that indicate this besides VariantDatum.
I'd appreciate any advice before taking the hammer or even getting as far
as submitting a patch.

Regards,
Botond

[Piotr Tabor]

While trying to use the code I ran into a few issues:

1. I like to compile my code with all compiler warnings enabled.
Unfortunately when including the supersonic headers, gcc produces a ton
of warnings. The following gcc switches need to be added to suppress
these: -Wno-shadow -Wno-conversion -Wno-unused-parameter
-Wno-switch-default -Wno-sign-compare -Wno-return-type
-Wno-ignored-qualifiers
Is there any reason you are not using these?

No other reason than the fact that we should fix this warnings instead of silencing them.  

 

2. I have a use-case where some data I need to operate on is
weakly-typed, similarly to how variables in perl and some other languages
are handled. e.g.  $var >= 42 would work for integers and also strings
that can be parsed as an int. The VariantDatum is mostly an ideal class
for this data,  unfortunately this is unused elsewhere in the code. It
would involve quite an amount of code to introduce it into the type
system and make most expressions/operations compatible with this.
I know that this wouldn't be terribly efficient due to the overhead
involved, contrary to the design goals of supersonic, but I can't really
think of anything else right now to be able to solve this problem simpler
and less intrusively.

There are no plans to introduce 'untyped' fields. You can consider having multiple fields: 

a_int, a_string, a_double and building your expression in such way that the desired semantic in 

obtained. Probably you could build a layer on top of supersonic that would hide this situation, 

i.e. You build an Expression  or Operation and its automatically bound to complicated 'bound'

expression that resolves the type-conversion problems.

Another half-baked solution was to use some kind of type-casting or
parser functions which would then explicitly define the return type of
the data, e.g. int($var) >= 42
The only case this wouldn't work for example is comparing two variables,
$var1 == $var2. The expression should evaluate without an error if the two
types are the same (or can be casted/parsed), whereas doing an explicit
type-casting would fail in the following case if the vars contain values
that don't look like integers: int($var1) == int($var2)

I don't see a reason to not have a function in supersonic that for given expression returns (as STRING)

the computed type of this expression. This way you could have expression: 

Something like:  IF( TYPE(a) = TYPE(b), STRING(a) == STRING(b), FALSE)

You could also write your own BoundExpression that do the comparison of two columns of any type. 

3. In addition to the above, I need to handle some custom data types
(complex structures, like list or map) which are returned in the result
view. This is similar to case 2 above, except that there is no need to
execute any expression evaluation on this data. This could be easily
solved by adding a custom DataType to supersonic.proto and types.h, but I
thought I'd ask in case there is a solution that could work without the
need to modify the code at all.

Well. You can hack around this problem. For example use STRING column (StringPiece is stored in the column). 

In standard scenarios this StringPiece points to some memory in Arena or other place that is interpreted as STRING value. 

 

But you could set the StringPiece size to 1 (I'm not sure if 0 is not used in a special way), and the pointer to the pointer to your custom data. 

You can also consider using INT64 instead...

 

 

I'm pretty sure you guys have already dealt with issues 2) and 3), there
are traces in the code that indicate this besides VariantDatum.

We are experimenting around adding more advanced structure handling (protobuf style data) to Supersonic. But its far future. 

 

I'd appreciate any advice before taking the hammer or even getting as far as submitting a patch.

Piotr 

[Botond Botyanszki]

Hi,

On Thu, 27 Feb 2014 18:03:54 +0100
Piotr Tabor <pt...@google.com> wrote:

> Another half-baked solution was to use some kind of type-casting or
> > parser functions which would then explicitly define the return type of
> > the data, e.g. int($var) >= 42
> > The only case this wouldn't work for example is comparing two variables,
> > $var1 == $var2. The expression should evaluate without an error if the two
> > types are the same (or can be casted/parsed), whereas doing an explicit
> > type-casting would fail in the following case if the vars contain values
> > that don't look like integers: int($var1) == int($var2)
> >
>
> I don't see a reason to not have a function in supersonic that for given
> expression returns (as STRING)
> the computed type of this expression. This way you could have expression:
>
> Something like: IF( TYPE(a) = TYPE(b), STRING(a) == STRING(b), FALSE)
>
> You could also write your own BoundExpression that do the comparison of two
> columns of any type.

I have also considered this approach, unfortunately there are problems
with this when the string representation is compared instead of the other
way around. For example the following should be all TRUE:
"1.0" == 1
"1" == 1.0

> 3. In addition to the above, I need to handle some custom data types
> > (complex structures, like list or map) which are returned in the result
> > view. This is similar to case 2 above, except that there is no need to
> > execute any expression evaluation on this data. This could be easily
> > solved by adding a custom DataType to supersonic.proto and types.h, but I
> > thought I'd ask in case there is a solution that could work without the
> > need to modify the code at all.
> >
>
> Well. You can hack around this problem. For example use STRING column
> (StringPiece is stored in the column).
> In standard scenarios this StringPiece points to some memory in Arena or
> other place that is interpreted as STRING value.
>
> But you could set the StringPiece size to 1 (I'm not sure if 0 is not used
> in a special way), and the pointer to the pointer to your custom data.
>
> You can also consider using INT64 instead...

I considered this. The problem is that there are other columns in the
result view which can hold real STRING or INT64 values and in this case
it is impossible to tell what the datatype is. Using the column name
could be an option, but this is also a little ugly. Looks like the
cleanest option is to add another datatype.

Thanks for your suggestions!

Regards,
Botond