Unlimited search results

[Sergey Abakumoff]

Hello guys,
Here is the demo of what I have faced in my project:

package main

import (
	"encoding/json"
	"flag"
	"fmt"
	"log"
	"os"

	"github.com/blevesearch/bleve"
	"github.com/blevesearch/bleve/mapping"
)

func buildMapping() mapping.IndexMapping {

	textFieldMapping := bleve.NewTextFieldMapping()
	textFieldMapping.Analyzer = "en"

	userMapping := bleve.NewDocumentMapping()
	userMapping.AddFieldMappingsAt("name", textFieldMapping)

	indexMapping := bleve.NewIndexMapping()
	indexMapping.TypeField = "Type"
	indexMapping.DefaultAnalyzer = "en"
	indexMapping.AddDocumentMapping("user", userMapping)

	return indexMapping
}

var indexPath = flag.String("index", "index.bleeve", "index path")

func main() {
	flag.Parse()
	defer os.RemoveAll(*indexPath)
	bleveIndex, err := bleve.Open(*indexPath)

	if err == bleve.ErrorIndexPathDoesNotExist {
		bleveIndex, err = bleve.New(*indexPath, buildMapping())
		if err != nil {
			log.Fatal(err)
		}
	}
	for i := 1; i < 150; i++ {
		data := json.RawMessage(`{"name" : "john", "Type" : "user"}`)
		var user interface{}
		if err := json.Unmarshal(data, &user); err != nil {
			panic(err)
		}

		if err = bleveIndex.Index(fmt.Sprintf("id#%[1]d", i), user); err != nil {
			panic(err)
		}
	}
	termQuery := bleve.NewTermQuery("john")
	termQuery.SetField("name")
	typeQuery := bleve.NewTermQuery("user")
	typeQuery.SetField("Type")
	query := bleve.NewConjunctionQuery(termQuery, typeQuery)

	searchRequest := bleve.NewSearchRequest(query)
	searchRequest.Highlight = bleve.NewHighlight()
	
	searchResults, err := bleveIndex.Search(searchRequest)

	if err != nil {
		panic(err)
	}

	fmt.Println(len(searchResults.Hits))
}

The output of this program is 10 while the total number of matching documents is 150! I am wondering what would be the proper way to tell search request to return all the matching documents. I know that I could do something like

searchRequest.Size = 100000

but it's very hackish solution. Do you have any suggestions on how to solve this problem?

[Sergey Abakumoff]

[Marty Schoch]

Currently there is no other solution, you just have to set the size to the total number of documents.

We do want to improve support for this in the future, but you've already opened an issue for it, and we will track progress there:

https://github.com/blevesearch/bleve/issues/640

marty

--
You received this message because you are subscribed to the Google Groups "bleve" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bleve+unsubscribe@googlegroups.com.
To post to this group, send email to bl...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bleve/6b4bace7-90dc-4866-874c-6ac0a4942e32%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sergey Abakumoff]

Thank you! Hope to get this improvement soon.

To unsubscribe from this group and stop receiving emails from it, send an email to bleve+un...@googlegroups.com.

[Sergey Abakumoff]

I have one more question..My app is using boltdb as the main data storage and it does not have a simple way to obtain the number of documents within a bucket..If I use the code like searchRequest.Size = math.MaxInt32 to make sure that all the matches will be returned, how hard the performance penalty will be if any?

On Monday, November 13, 2017 at 8:38:46 PM UTC+1, Marty Schoch wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to bleve+un...@googlegroups.com.