Why Go wrapper for YottaDB so slow?

[Сергей Каменев]

Hi!

Inserts through the Go wrapper are about 10 times slower comparing with M.
By inserting 1M of values.

time ./speedYotta

real 0m13,728s
user 0m18,737s
sys 0m0,914s

time yottadb -run ^speedYotta

real 0m1,617s
user 0m1,326s
sys 0m0,278s

time ./sY1

real 0m13,075s
user 0m18,786s
sys 0m0,751s

Source codes:
=== speedYotta.go ===
-------------------
package main

import (
"lang.yottadb.com/go/yottadb"
"strconv"
)

func main() {

defer yottadb.Exit()

s := ""
for i := 1; i < 1000000; i++ {
s = strconv.Itoa(i)
yottadb.SetValE(yottadb.NOTTP, nil, s, "^hello", []string{s})
}
}
-------------------------

=== speedYotta.m ===
---------------------------
speedYotta

for i=0:1:1000000 do
. set ^a(i)=i
---------------------------

=== sY1.go ===
---------------------------
package main

import (
"lang.yottadb.com/go/yottadb"
"strconv"
)

func main() {
defer yottadb.Exit()

s := strconv.Itoa(5)
for i := 1; i < 1000000; i++ {
yottadb.SetValE(yottadb.NOTTP, nil, s, "^hello", []string{s})
}
}
---------------------------

[K.S. Bhaskar]

There are two reasons. The biggest one is that CGO is heavyweight. A much smaller one is the way Go does garbage collection emphasizes predictable response time, the way that YottaDB does garbage collection is to minimize CPU usage. You are basically doing what I'd call a “point” benchmark. In the context of a complete application, where an application executes business logic interspersed with database accesses, you are likely to find different performance characteristics. For example, Go's applications using Goroutines may execute business logic faster than M's single flow-of-control code execution.

Regards
– Bhaskar

[Сергей Каменев]

Is it potentially possible to increase the speed of the CGO wrapper?

Sergey.

[K.S. Bhaskar]

There has been much discussion of CGO overhead on the golang-nuts discussion group.

Incidentally, you are using the YottaDB Go Easy API. You may find the Go Simple API to be a bit faster, although I don't know how much faster it will be for you.

Regards
– Bhaskar

[Сергей Каменев]

Thank you, Bhaskar!

Yes, Go Simple API 3 times faster comparing with Go Easy API.
And Go Simple API 3 times slower comparing with native M code for inserting.

time ./speedYSimple

real 0m4,391s
user 0m4,250s
sys 0m0,570s

If we move some operations per cycle, then

time ./speedYSimple1

real 0m3,425s
user 0m3,296s
sys 0m0,458s

=== Code of speedYSimple.go: =====

package main

import (
"lang.yottadb.com/go/yottadb"
"strconv"
)

const maxglnamelen uint32 = 8
const maxglsubscount uint32 = 1
const maxglsubslen uint32 = 10 //128
const tptoken uint64 = yottadb.NOTTP

func main() {
var glname yottadb.KeyT
var errstr, value yottadb.BufferT

value.Alloc(maxglsubslen)
errstr.Alloc(yottadb.YDB_MAX_ERRORMSG)
glname.Alloc(maxglnamelen, maxglsubscount, maxglsubslen)

defer yottadb.Exit()

defer glname.Free()
defer errstr.Free()
defer value.Free()

s := ""
for i := 1; i < 1000000; i++ {
s = strconv.Itoa(i)

glname.Varnm.SetValStr(tptoken, &errstr, "^hello")
glname.Subary.SetElemUsed(tptoken, &errstr, maxglsubscount)
glname.Subary.SetValStr(tptoken, &errstr, 0, s)

value.SetValStr(tptoken, &errstr, s)

glname.SetValST(tptoken, &errstr, &value)
}
}


=== Code of speedYSimple1.go: =====

package main

import (
"lang.yottadb.com/go/yottadb"
"strconv"
)

const maxglnamelen uint32 = 8
const maxglsubscount uint32 = 1
const maxglsubslen uint32 = 10 //128
const tptoken uint64 = yottadb.NOTTP

func main() {
var glname yottadb.KeyT
var errstr, value yottadb.BufferT

value.Alloc(maxglsubslen)
errstr.Alloc(yottadb.YDB_MAX_ERRORMSG)
glname.Alloc(maxglnamelen, maxglsubscount, maxglsubslen)

defer yottadb.Exit()

defer glname.Free()
defer errstr.Free()
defer value.Free()

glname.Varnm.SetValStr(tptoken, &errstr, "^hello")
glname.Subary.SetElemUsed(tptoken, &errstr, maxglsubscount)

s := ""
for i := 1; i < 1000000; i++ {
s = strconv.Itoa(i)

glname.Subary.SetValStr(tptoken, &errstr, 0, s)

value.SetValStr(tptoken, &errstr, s)

glname.SetValST(tptoken, &errstr, &value)
}
}

[K.S. Bhaskar]

Thanks for the update. I'm glad you found the simple API to be faster.

Regards
– Bhaskar

[Сергей Каменев]

Very good speed and very simple interface have "mg_go" extension.
https://github.com/chrisemunt/mg_go

I'm not sure if he knows how to handle Go-multithreading and parallel transactions correctly. But using it is very simple and pleasant.

time ./speedYMG

real 0m3,196s
user 0m3,069s
sys 0m0,326s

=== Code speedYMG.go ===
package main

import (
"mg_go"
)

func main() {
db := mg_go.New("YottaDB")
db.APImodule = "/usr/local/lib64/mg_dba.so"
db.Path = "/opt/yottadb/r1.32"
db.EnvVars = db.EnvVars + "ydb_dir=/home/inetstar/.yottadb\n"
db.EnvVars = db.EnvVars + "ydb_rel=r1.32_x86_64\n"
db.EnvVars = db.EnvVars + "ydb_gbldir=/home/inetstar/.yottadb/r1.32_x86_64/g/yottadb.gld\n"
db.EnvVars = db.EnvVars + "ydb_routines=/home/inetstar/.yottadb/r1.32_x86_64/o*(/root/.yottadb/r1.32_x86_64/r /home/inetstar/.yottadb/r) /opt/yottadb/r1.32/libyottadbutil.so\n"
db.EnvVars = db.EnvVars + "ydb_ci=/opt/yottadb/r1.32/zmgci.ci\n"
db.EnvVars = db.EnvVars + "\n"

db.Open()

defer db.Close()

gHello := db.Global("^hello")

for i := 1; i < 1000000; i++ {

gHello.Set(i, i)
}
}

[K.S. Bhaskar]

That's good news, and if that works for you, please use it. Chris Munt does good work. The only thing to check is to make sure that it works well with Goroutines if your application uses them, as most applications do.

Regards
– Bhaskar