Weird problem with stored and UTF-8 :-(

[Adam]

Hi Maxime,
Hi all !

I'm using DPS a little bit since couple of years and I really love
it !!
I have small search engine which worked very fine although under very
small load since couple of years,
it was last compiled with a snapshot dated 4.53-31122009 and
everything was setup with ISO-8859-1 charset including mysql working
also in latin1 encoding.

Recently I have been obliged to migrate everything to another server
working with mysql5 and full unicode.
So I have taken the latest snapshot dpsearch-4.54-2012-04-15-mysql and
decided to go with full unicode.
I didnt import anything from the old server, only my url.txt and
looked on my old configs a bit to setup the new ones: the indexing has
been started from the scratch using brand new empty db and indexing
process is working fine.

Here are my most important parameters :

[mysqld]
character-set-server=utf8
collation-server=utf8_general_ci

At the end of each DBAddr command, I have added "&charset=utf8"

in my indexer.conf :
LocalCharset UTF-8

in my searchd.conf :
LocalCharset UTF-8

in my search.htm :
LocalCharset UTF-8
BrowserCharset UTF-8
Locale fr_FR.UTF-8

I'm using also stored.conf but there is no LocalCharset UTF-8 in this
file.

And stored behaviour seems to be the problem !
Once searchd is trying to connect to stored everything becomes
frozen :

Apr 15 15:08:04 server2 search.cgi[15142]: {00} Start Order, Last-
Modified and Excerpts
Apr 15 15:08:04 server2 search.cgi[15142]: {00} Stop Order, Last-
Modified and Excerpts: 0,00
Apr 15 15:08:04 server2 search.cgi[15142]: {00} Start DpsTrack
Apr 15 15:08:04 server2 search.cgi[15142]: {00} Stop DpsTrack: 0,00
Apr 15 15:08:04 server2 search.cgi[15142]: {00} Done DpsFind 0,005
Apr 15 15:08:47 server2 search.cgi[15189]: {00} search.cgi started
with '/usr/local/dpsearch/etc/search.htm'
Apr 15 15:08:47 server2 search.cgi[15189]: {00} VarDir: '/home/var'
Apr 15 15:08:47 server2 search.cgi[15189]: {00} Affixes: 559, Spells:
122605, Synonyms: 6412, Acronyms: 7, Stopwords: 164
Apr 15 15:08:47 server2 search.cgi[15189]: {00} Chinese dictionary
with 0 entries
Apr 15 15:08:47 server2 search.cgi[15189]: {00} Korean dictionary with
0 entries
Apr 15 15:08:47 server2 search.cgi[15189]: {00} Thai dictionary with 0
entries
Apr 15 15:08:47 server2 search.cgi[15189]: {00} Start DpsFind
Apr 15 15:08:47 server2 search.cgi[15189]: {00} DpsFind for searchd://localhost/
Apr 15 15:08:47 server2 search.cgi[15189]: {00} DpsGetWords for
searchd://localhost/
Apr 15 15:08:47 server2 searchd[14898]: {300} Query:
[192.168.1.27:search.htm:UTF-8:] Asie
Apr 15 15:08:47 server2 stored[14899]: {00} [127.0.0.1] DpsBaseOpen
error: rec_id: 2b15a2c9
Apr 15 15:08:47 server2 stored[14899]: {00} Retrieve by chunks error
Apr 15 15:08:47 server2 stored[14899]: {00} [127.0.0.1] DpsBaseOpen
error: rec_id: ed4bf072
Apr 15 15:08:47 server2 stored[14899]: {00} Retrieve by chunks error
Apr 15 15:08:47 server2 stored[14899]: {00} [127.0.0.1] DpsBaseOpen
error: rec_id: 1b5f4257
Apr 15 15:08:47 server2 stored[14899]: {00} Retrieve by chunks error
Apr 15 15:08:47 server2 stored[14899]: {00} [127.0.0.1] DpsBaseOpen
error: rec_id: cfa98fb0
Apr 15 15:08:47 server2 stored[14899]: {00} Retrieve by chunks error
Apr 15 15:08:47 server2 stored[14899]: {00} [127.0.0.1] DpsBaseOpen
error: rec_id: e53e2524
Apr 15 15:08:47 server2 stored[14899]: {00} Retrieve by chunks error
Apr 15 15:08:47 server2 stored[14899]: {00} [127.0.0.1] DpsBaseOpen
error: rec_id: 32ca3345
Apr 15 15:08:47 server2 stored[14899]: {00} Retrieve by chunks error
Apr 15 15:08:47 server2 stored[14899]: {00} Sun 15 15:08:47 [14899]
<hello> error 2 €
Apr 15 15:08:47 server2 stored[14899]: {00} Sun 15 15:08:47 [14899]
<hello> error 2 V €
Apr 15 15:08:47 server2 stored[14899]: {00} Sun 15 15:08:47 [14899]
<hello> error 2 @ €
Apr 15 15:08:47 server2 stored[14899]: {00} Sun 15 15:08:47 [14899]
<hello> error 2 8 €
Apr 15 15:08:47 server2 stored[14899]: {00} Sun 15 15:08:47 [14899]
<hello> error 2 £ €
Apr 15 15:08:47 server2 stored[14899]: {00} [127.0.0.1] DpsBaseOpen
error: rec_id: dad32bef
Apr 15 15:08:47 server2 stored[14899]: {00} Retrieve by chunks error


And stored is hunged, no response at all for any search through
searchd deamon;
Then after indexing couple of pages indexer has refreshed everything
because
it was possible to perform 2 searches without restarting all deamons:


Apr 15 16:23:48 server2 indexer[19610]: {00} url data and limits Done
Apr 15 16:23:48 server2 cached[3138]: {201} url data and limits Done
Apr 15 16:23:48 server2 cached[3138]: {201} Sun 15 16:23:48 [3138]
Flushing all buffers...
Apr 15 16:23:48 server2 indexer[19610]: {00} Cached buffers flush Done
Apr 15 16:23:48 server2 cached[3138]: {201} Done
Apr 15 16:23:48 server2 cached[3138]: {201} Sun 15 16:23:48 [3138]
Client action BYE received.
Apr 15 16:23:48 server2 searchd[14897]: {00} Query Tracker: SIGTERM
arrived
Apr 15 16:23:48 server2 searchd[3141]: {00} URL data preloaded. 4480
bytes of memory used
Apr 15 16:23:48 server2 searchd[3141]: {00} Ready
Apr 15 16:23:48 server2 stored[19629]: {00} [127.0.0.1] Connected.
PORT: 158,252
Apr 15 16:23:48 server2 searchd[19628]: {300} Query:
[192.168.1.27:search.htm:UTF-8:] Thaïlande
Apr 15 16:23:48 server2 searchd[19628]: {300} SIGPIPE arrived. Broken
pipe!
Apr 15 16:23:48 server2 searchd[19627]: {00} Query Tracker: SIGTERM
arrived
Apr 15 16:23:48 server2 searchd[3141]: {00} URL data preloaded. 4480
bytes of memory used
Apr 15 16:23:48 server2 searchd[3141]: {00} Ready
Apr 15 16:23:48 server2 stored[19632]: {00} [127.0.0.1] Connected.
PORT: 232,182
Apr 15 16:23:48 server2 searchd[19631]: {300} Query:
[192.168.1.27:search.htm:UTF-8:] Italie
Apr 15 16:23:48 server2 searchd[19631]: {300} SIGPIPE arrived. Broken
pipe!
Apr 15 16:23:49 server2 stored[19634]: {00} [127.0.0.1] Connected.
PORT: 147,141
Apr 15 16:23:49 server2 searchd[19633]: {300} Query:
[192.168.1.27:search.htm:UTF-8:] Asie
Apr 15 16:23:49 server2 searchd[19633]: {300} SIGPIPE arrived. Broken
pipe!
Apr 15 16:23:49 server2 stored[19636]: {00} [127.0.0.1] Connected.
PORT: 148,180
Apr 15 16:23:49 server2 searchd[19635]: {300} Query:
[192.168.1.27:search.htm:UTF-8:] Italie
Apr 15 16:23:49 server2 stored[19636]: {00} [127.0.0.1] DpsBaseOpen
error: rec_id: 4f282fb9
Apr 15 16:23:49 server2 stored[19636]: {00} Retrieve by chunks error
Apr 15 16:23:49 server2 stored[19636]: {00} [127.0.0.1] DpsBaseOpen
error: rec_id: b57d3da7
Apr 15 16:23:49 server2 stored[19636]: {00} Retrieve by chunks error
Apr 15 16:23:49 server2 search.cgi[19549]: {00} Start Order, Last-
Modified and Excerpts
Apr 15 16:23:49 server2 search.cgi[19549]: {00} Stop Order, Last-
Modified and Excerpts: 0,00
Apr 15 16:23:49 server2 search.cgi[19549]: {00} Start DpsTrack
Apr 15 16:23:49 server2 search.cgi[19549]: {00} Stop DpsTrack: 0,00
Apr 15 16:23:49 server2 search.cgi[19549]: {00} Done DpsFind 72,090
Apr 15 16:24:05 server2 search.cgi[19654]: {00} search.cgi started
with '/usr/local/dpsearch/etc/search.htm'
Apr 15 16:24:05 server2 search.cgi[19654]: {00} VarDir: '/home/var'
Apr 15 16:24:05 server2 search.cgi[19654]: {00} Affixes: 559, Spells:
122605, Synonyms: 6412, Acronyms: 7, Stopwords: 164
Apr 15 16:24:05 server2 search.cgi[19654]: {00} Chinese dictionary
with 0 entries
Apr 15 16:24:05 server2 search.cgi[19654]: {00} Korean dictionary with
0 entries
Apr 15 16:24:05 server2 search.cgi[19654]: {00} Thai dictionary with 0
entries
Apr 15 16:24:05 server2 search.cgi[19654]: {00} Start DpsFind
Apr 15 16:24:05 server2 search.cgi[19654]: {00} DpsFind for searchd://localhost/
Apr 15 16:24:05 server2 search.cgi[19654]: {00} DpsGetWords for
searchd://localhost/
Apr 15 16:24:05 server2 searchd[19635]: {300} Query:
[192.168.1.27:search.htm:UTF-8:] Italie
Apr 15 16:24:05 server2 stored[19636]: {00} [127.0.0.1] DpsBaseOpen
error: rec_id: b57d3da7
Apr 15 16:24:05 server2 stored[19636]: {00} Retrieve by chunks error
Apr 15 16:24:05 server2 stored[19636]: {00} [127.0.0.1] DpsBaseOpen
error: rec_id: 4f282fb9
Apr 15 16:24:05 server2 stored[19636]: {00} Retrieve by chunks error
Apr 15 16:24:05 server2 search.cgi[19654]: {00} Start Order, Last-
Modified and Excerpts
Apr 15 16:24:05 server2 search.cgi[19654]: {00} Stop Order, Last-
Modified and Excerpts: 0,00
Apr 15 16:24:05 server2 search.cgi[19654]: {00} Start DpsTrack
Apr 15 16:24:05 server2 search.cgi[19654]: {00} Stop DpsTrack: 0,00
Apr 15 16:24:05 server2 search.cgi[19654]: {00} Done DpsFind 0,005
Apr 15 16:24:08 server2 clamd[4189]: SelfCheck: Database status OK.
Apr 15 16:24:39 server2 search.cgi[19657]: {00} search.cgi started
with '/usr/local/dpsearch/etc/search.htm'
Apr 15 16:24:39 server2 search.cgi[19657]: {00} VarDir: '/home/var'
Apr 15 16:24:39 server2 search.cgi[19657]: {00} Affixes: 559, Spells:
122605, Synonyms: 6412, Acronyms: 7, Stopwords: 164
Apr 15 16:24:39 server2 search.cgi[19657]: {00} Chinese dictionary
with 0 entries
Apr 15 16:24:39 server2 search.cgi[19657]: {00} Korean dictionary with
0 entries
Apr 15 16:24:39 server2 search.cgi[19657]: {00} Thai dictionary with 0
entries
Apr 15 16:24:39 server2 search.cgi[19657]: {00} Start DpsFind
Apr 15 16:24:39 server2 search.cgi[19657]: {00} DpsFind for searchd://localhost/
Apr 15 16:24:39 server2 search.cgi[19657]: {00} DpsGetWords for
searchd://localhost/
Apr 15 16:24:39 server2 searchd[19635]: {300} Query:
[192.168.1.27:search.htm:UTF-8:] Venise
Apr 15 16:24:39 server2 stored[19636]: {00} [127.0.0.1] DpsBaseOpen
error: rec_id: 890ae945
Apr 15 16:24:39 server2 stored[19636]: {00} Retrieve by chunks error
Apr 15 16:24:39 server2 stored[19636]: {00} Sun 15 16:24:39 [19636]
<hello> error 2 ‹ €
Apr 15 16:24:39 server2 stored[19636]: {00} [127.0.0.1] DpsBaseOpen
error: rec_id: dad32bef
Apr 15 16:24:39 server2 stored[19636]: {00} Retrieve by chunks error
Apr 15 16:24:39 server2 stored[19636]: {00} [127.0.0.1] DpsBaseOpen
error: rec_id: 2032ca33
Apr 15 16:24:39 server2 stored[19636]: {00} Retrieve by chunks error
Apr 15 16:24:39 server2 stored[19636]: {00} Sun 15 16:24:39 [19636]
<hello> error 2 § €
Apr 15 16:24:39 server2 stored[19636]: {00} Sun 15 16:24:39 [19636]
<hello> error 2 = €
Apr 15 16:24:39 server2 stored[19636]: {00} Sun 15 16:24:39 [19636]
<hello> error 2 } €
Apr 15 16:24:39 server2 stored[19636]: {00} Sun 15 16:24:39 [19636]
<hello> error 2 µ €

What could be the heck ? ;)
I'm fighting with this issue since yesterday and now I have to give up
for the moment :)
I have searched the forum for 'DpsBaseOpen error" and "stored" but
didnt find anything :(
Thanks in advance for any clues :)

Adam

PS:
Also, Im not sure about one detail: if you work with full unicode
everywhere, should all config files be also converted to UTF-8 or stay
in latin1 ??
Obviously search.htm is UTF-8 encoded but my config files stay in
latin1.

[Maxim Zakharov]

Hi Adam,

If you use searchd and stored on the very same PC, you can configure
searchd to operate with stored database directly, without stored.
To do so, you need to remove stored connection from DBAddr command in
searchd.conf and put the following commands in it:

DoStore yes
StoredFiles 0x300

the value of StoredFiles must be the same as in your stored.conf file.

Maxim

--
http://www.dataparksearch.org/ - an open source search engine.

[Adam]

Many thanks Maxime for your clarification :)
In fact as I have everything on the same server, I have switched already to "DoStore yes" option in indexer.conf and therefore stopped the stored deamon; in this configuration everything is working without any problems or errors. Also adding "&charset=utf8" to all configurations has solved all my problems with some caracters encoding.

However according to some posts at this forum the solution with stored deamon would be much faster, so I would like to try to implement it anyway.

Could you please just confirm me if the following actions and configurations seems right to you ?

=> indexer.conf
DBAddr mysql://xxx:XXX@localhost/search/?dbmode=cache&cached=localhost&charset=utf8&stored=localhost
DoStore yes
WrdFiles 4096

=> searchd.conf
DBAddr  mysql://xxx:XXX@localhost/search/?dbmode=cache&charset=utf8&trackquery&stored=localhost
DoStore yes
StoredFiles 0x300
WrdFiles 4096

=> search.htm
DBAddr searchd://localhost
DoStore yes

=> stored.conf
DBAddr mysql://xxx:XXX@localhost/search/?dbmode=cache&charset=utf8
StoredFiles 0x300
WrdFiles 4096

=> cached.conf
DBAddr  mysql://xxx:XXX@localhost/search/?dbmode=cache&charset=utf8
DoStore yes
StoredFiles 0x300
WrdFiles 4096

So I will restart stored deamon and you can notice that I dont have "StoredFiles 0x300" in my indexer.conf
Is it all OK ?

Many thanks in advance.

Adam