[ANN] Your Golang powered NoSQL DB engine tiedot version 1.1 is here!

[Howard Guo]

Hello Gophers!

tiedot is your Golang (100%) powered document database engine that uses JSON for documents and queries; it can be embedded into your program, or run a stand-alone server using HTTP for an API.

Since the initial tiedot release 1.0 (over 3 months ago), many feature and stability improvements have been made, and today I would like to present to you version 1.1!

Repository:

https://github.com/HouzuoGuo/tiedot

Highlights:

- New query syntax - easier and more efficient

- Cross-platform - tiedot runs well on *nix and Windows

- Reduced memory consumption

- Improved scalability

I hope you enjoy, and please leave your valuable feedback/comments!

Kind regards,

Howard

[Tom D]

Just thought I would throw in my two cents.

I am using tiedot for some of my own projects; its pretty damn good. It outperforms sqlite by quite a margin, and seems to scale (even with massive amounts of data) really well, maintaining the benchmarks displayed on the README.

One such project involved inserting the entire contents of english Wikipedia articles into a tiedot database, so that my meshnet could access wikipedia without connecting to the public internet. It gobbled up FIVE POINT NINE MILLION ARTICLES OF PURE TERROR in just over an hour. No corruption issues. No scale issues. Lookups and fetches still ridiculously fast.

I am also using this engine as the backend of a web application in production.

If you are after a high-speed pure-go database engine that you can embed in your project, tiedot is for you.

[Fino]

hi howard,

I was comparing  MongoDB, tiedot, neo4j last week  for my little  android app's backend,  finally I want to try neo4j because it looks more fancy,

how do u think about Graphic data base ,  compare with  JSON style nosql database?

thanks,

fino

在 2013年10月8日星期二UTC+8上午7时41分17秒，Howard Guo写道：

[Howard Guo]

Hello Fino.

I cannot comment on the features and usages of graph database due to lack of relevant working experience, sorry.

I look forward to Go being an Android programming language one day...

Regards,

Howard

[Dobrosław Żybort]

Tiedot looks promising, will check it when I have some free time.

Would be great to have something like ArangoDB[1] or OrientDB[2] written in Go. They are multi model databases (key-value/document/graph and document/graph) and both of them use they document storage to store graph relations.

Best regards,
Dobrosław Żybort

[1] http://www.arangodb.org/
[2] http://www.orientdb.org/

[mattn]

Bram, why you remove this patch from todo list?

https://code.google.com/p/vim/source/diff?spec=svn173c9c860e42e2af31d598aa6924b99d7e73bd1b&r=173c9c860e42e2af31d598aa6924b99d7e73bd1b&format=side&path=/runtime/doc/todo.txt

[mattn]

Oooops, sorry, I mistaken to reply.

[paulo....@gmail.com]

Not sure if it will ever happen,

http://code.google.com/p/android/issues/detail?id=39482&q=paulo&colspec=ID%20Type%20Status%20Owner%20Summary%20Stars

[guo tie]

Good job.

I hope it can support Graphic database

[Kevin P]

Any plans on adding RPC support, either by protobuf or GOB?

[Tom D]

There is a fully-featured HTTP RPC, which is documented on the wiki, but no protobuf/gob version. It should be trivial to make requests to the database using net/http calls.

[Kyle Lemons]

HTTP adds a lot of extra bytes and introduces a nontrivial amount of processing overhead compared to net/rpc.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Howard Guo]

Thanks Kyle.

After taking a look at Go RPC support, it seems very convenience and sounds like a very good programming exercise!

https://github.com/HouzuoGuo/tiedot/issues/26

[euric...@gmail.com]

Hi! I have a question about Tiedot queries. How do you query the content of an array of structures? I have this program:

// main

package main

import (

"encoding/json"

"fmt"

"loveoneanother.at/tiedot/db"

"strconv"

)

type Arr struct {

A string `json:"A"`

B int    `json:"B"`

}

type Str struct {

C int   `json:"C"`

D []Arr `json:"D"`

}

func main() {

myDB, err := db.OpenDB("C:\\tmp\\MyDatabase")

defer myDB.Close()

if err != nil {

panic(err)

}

myDB.Drop("A")

myDB.Create("A")

col := myDB.Use("A")

arr1 := Arr{A: "A", B: 1}

arr2 := Arr{A: "a", B: 0}

arr := []Arr{arr1, arr2}

str := Str{C: 2, D: arr}

fmt.Println(str)

var q int

q = 1

fmt.Println("Searching for", q)

_, err2 := col.Insert(str)

if err2 != nil {

panic(err2)

}

result := make(map[uint64]struct{})

var query interface{}

x := `{"eq":{"B":` + strconv.Itoa(q) + `},"in": ["C"]}`

//x := `"all"`

e := json.Unmarshal([]byte(x), &query)

if e != nil {

fmt.Println("Json error:", e)

} else {

fmt.Println("query=", query)

}

e2 := db.EvalQueryV2(query, col, &result)

if e2 != nil {

fmt.Println("Query error:", e2)

} else {

fmt.Println("Proceeding...")

fmt.Println("Result=", result)

}

if result == nil {

fmt.Println("Found nothing")

} else {

for id := range result {

fmt.Println("Found:", id)

var z map[string]interface{}

col.Read(id, &z)

fmt.Println(z)

}

}

fmt.Println("Hello World!")

}

How do I query the contents of D?

[j...@langevin.me]

Does tiedot support ordering results? If so, example query?

Wanting to combine w/ limit to get the highest priority result, would rather not have to fetch entire result set to perform ordering on client.

-Jon

On Monday, October 7, 2013 7:41:17 PM UTC-4, Howard Guo wrote:

[Howard Guo]

Hello Jon.

In the current version, tiedot supports result ordering, although it
is somewhat limited - tiedot can optimize result of discrete integer
lookup queries.

So far, tiedot has only one type of index - hash table index. Integer
range lookup works by calculating hash table lookup results, and
combine them together. An example use case is to find documents
belonging to a specific day of month, or day of week. The result is
ordered on server.

I am making progress on making another index (skip list) to support
more general range queries and ordering.

Regards, Howard

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "golang-nuts" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/golang-nuts/5MalhKD6fjg/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

[Jonathan Langevin]

Is ordering automatically applied or does an argument have to be specified to get that behavior?

Using the embedded server, so I'm making direct function calls.

(sorry, resending so group is included)

[Howard Guo]

Hi Jon.

For example - there is an integer field "month" in documents; we would like to get month of the highest 100 records.

The range to scan is from 1 - 12, but because we want to get the "highest", therefore the range is reversed to 12 - 1. The query can be written as:

{"in": ["month"], "int-from": 12, "int-to": 1, "limit": 100}

tiedot scans documents for month == 12, collect documents, then scans for month == 11, collect documents and keep going until 100 documents have been collected.

This query feature is mentioned in https://github.com/HouzuoGuo/tiedot/wiki/API-V3-Reference

Regards,

Howard

[Jonathan Langevin]

Howard, do you have an API/doc reference for the embedded usage scenario? Method arguments aren't really documented it seems, which currently forces others to have to read the source directly. Perhaps that's Go convention, dunno, I'm relatively new...

[Tom D]

http://godoc.org/github.com/HouzuoGuo/tiedot/db for documentation.

https://github.com/HouzuoGuo/tiedot/blob/master/example.go for example.

Post if you have any issues.

Best Regards,

me

[Jonathan Langevin]

Right, there's no mention anywhere of what each of these function arguments are expected to do, for instance the V2IntRange function has an "intFrom" arg, but what's it intended for? Seems similar to what would be provided in the "expr" arg (based on initial skim of the source).

V2IntRange(intFrom interface{}, expr map[string]interface{}, src *Col, result *map[uint64]struct{})

[Howard Guo]

Hey Jon.

I apologize for not having comprehensive embedded usage guide.

In that particular case:

V2IntRange can be called manually, but it is usually called by using V2Query.

Parameter intFrom is the JSON value of "int-from", expr is the entire integer range scan expression (for example {"int-from": 1, "int-to": 10, "in": ["something"]}), and result is pointer to a map where query result (document IDs) are put.

There's an issue raised for it - I shall extend and improve embedded usage examples.

If you could email me your use cases, I can compose a list of functions explain them - hopefully being useful to you.

Regards, Howard

[Howard Guo]

Hey Jonathan.

I have re-written the embedded usage examples, please check it out.

https://github.com/HouzuoGuo/tiedot/blob/master/example.go

And is there anything else I may help?

Regards,

Howard

[Jonathan Langevin]

That's excellent, I'll check it out.
Thanks!

[Howard Guo]

Hey Abhi.

I appreciate your interest in tiedot project.

To construct query filter on multiple document attributes, please use intersection query:

https://github.com/HouzuoGuo/tiedot/wiki/Query-processor-and-index#query-processor---supported-operations

Attribute filters are evaluated individually. The benchmark result (80k lookups per second) runs 3 attribute filters.

Query results return in the form of document IDs, you may retrieve these documents and extract the attributes you need.

Regards,

Howard

On 22 October 2015 at 09:26, <desaia...@gmail.com> wrote:

hi Howard

My requirement is - base on multiple fields, want to select value e.g

where =>

fromdate, todate, field1str, field2int 

select =>

rate, currency

can you please help me how tiedot gonann help me, also if any performance hit..

Rgds,

Abhi

[desaia...@gmail.com]

[desaia...@gmail.com]

Hi Howard

Thanks very much for the reply - I like your db as embedded for go lang compare to bolt and ledisdb

Can you please check if I done things correctly as it's working fine but slows down compare to single filed search

       >>creation 

       db_mydb = myDB.Use("mydb")

if err := db_mydb.Index([]string{"field1"}); err != nil {

panic(err)

}

if err := db_mydb.Index([]string{"field2"}); err != nil {

panic(err)

}

      >>200,0000 elements

      db_mydb.Insert(map[string]interface{}{"field1": "value", "field2": "value"...

     

     >> value := have unique values

     strq := `[{"n" : [{"in" : ["field1"], "eq":"value"}, {"in" : ["field2"], "eq":"value"}]}]`

     var query interface{}

     json.Unmarshal([]byte(strq), &query)

     queryResult := make(map[int]struct{})

     if err := db.EvalQuery(query, db_mydb, &queryResult); err != nil {

panic(err)

     }

     for id := range queryResult {

if err != nil {

           panic(err)

}

      }

Thanks very much

Rgds,

Abhi