Thanks for reply Marko,
Sure! I think the approach of making validation smarter and generating code for primitive types was the key.
Actually when you ask me for benchmark the first time, the performance was terrible!
If we have a look at how operation Filter is implemented:
Filter operation
We find 3 files (this is basically the same for all the operations)
In the filter.go, my input is a reflection type and obviously I am required to do a validation in order to avoid an uncontrolled panic
I guess my validation function is smarter than other libraries, Why? Because of I cache some info
func (op *Filter) validate() (*filterInfo, *errors.Error) {
item := &filterInfo{}
fnType := reflect.TypeOf(op.Func)
if val := cache.get(op.ItemsType, fnType); val != nil {
return val, nil
//...... Validations for input
item.fnInputType = fnIn.Type()
cache.add(op.ItemsType, fnType, item)
return item, nil
}
By this "smart cache" I avoid to evaluate eternally the same things. I mean,
If my stream contains string's and the filter func looks like func(string)bool I only evaluate once
because the output will be always the same
The dispatcher.go has been generated. This has been generated from repository
koazee-gen
What do I achieve with this generated code? That my operation are applied over a function that will work with primitive values instead of doing it with reflection
For example,
func filterString(itemsValue reflect.Value, function interface{}) interface{} {
input := itemsValue.Interface().([]string)
output := make([]string, 0)
fn := function.(func(string) bool)
for i := 0; i < len(input); i++ {
if fn(input[i]) {
output = append(output, input[i])
}
}
return output
}
Since I have a function for any primitive type (or pointers too) the performance doesn't suffer the reflection cost
I've also put a lot of effort to find the best way to do any operation with the best performance.
For example, for reverse operation I could do
func reverseInt16Ptr(itemsValue reflect.Value) interface{}{
input := itemsValue.Interface().([]*int16)
len := len(input)
output := make([]*int16, len)
for index := 0; index < len(input); index++ {
output[index]= input[len-1-index]
}
return output
}
but my generated function looks like below
func reverseInt16Ptr(itemsValue reflect.Value) interface{}{
input := itemsValue.Interface().([]*int16)
len := len(input)
output := make([]*int16, len)
for index := 0; index < (len/2)+1; index++ {
output[index], output[len-1-index] = input[len-1-index], input[index]
}
return output
}
I think is the best approach. I reduce a half the time of elements to be evaluated
I am learning a lot of Go (and enjoying too) with all your requests and suggestions and to be honest is so valuable when you achieve good results.
On roadmap , I would like to publish a richer set of operations for v0.0.3 (I know my provided set of operations is poor) and for v0.0.4 find a way to improve the performance for complex structs (I got a idea..)