make-index crashes

[Martin Pradny]

Hello,

 I tried to create index, but paster-make index crashed. I tried both with -f and without, but it made no difference. I had some index built when I installed RhodeCode month ago, but I don't remember wheter it crashed too.

Now I'm on 1.7.1

Any Ideas?

(rhodecode-venv)[scm@source rhodecode]$ paster make-index production.ini -f
2013-07-03 12:02:36.510 INFO  [rhodecode.model] initializing db for sqlite:////opt/rhodecode/rhodecode.db
2013-07-03 12:02:36.680 INFO  [rhodecode.model.scm] scanning for repositories in /home/scm
Traceback (most recent call last):
  File "/opt/rhodecode-venv/bin/paster", line 8, in <module>
    load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
  File "/opt/rhodecode-venv/lib/python2.6/site-packages/paste/script/command.py", line 104, in run
    invoke(command, command_name, options, args[1:])
  File "/opt/rhodecode-venv/lib/python2.6/site-packages/paste/script/command.py", line 143, in invoke
    exit_code = runner.run(args)
  File "/opt/rhodecode-venv/lib/python2.6/site-packages/rhodecode/lib/utils.py", line 720, in run
    return super(BasePasterCommand, self).run(args[1:])
  File "/opt/rhodecode-venv/lib/python2.6/site-packages/paste/script/command.py", line 238, in run
    result = self.command()
  File "/opt/rhodecode-venv/lib/python2.6/site-packages/rhodecode/lib/paster_commands/make_index.py", line 82, in command
    .run(full_index=self.options.full_index)
  File "/opt/rhodecode-venv/lib/python2.6/site-packages/rhodecode/lib/indexers/daemon.py", line 439, in run
    self.build_indexes()
  File "/opt/rhodecode-venv/lib/python2.6/site-packages/rhodecode/lib/indexers/daemon.py", line 424, in build_indexes
    self.index_files(file_idx_writer, repo_name, repo)
  File "/opt/rhodecode-venv/lib/python2.6/site-packages/rhodecode/lib/indexers/daemon.py", line 253, in index_files
    i, iwc = self.add_doc(file_idx_writer, idx_path, repo, repo_name)
  File "/opt/rhodecode-venv/lib/python2.6/site-packages/rhodecode/lib/indexers/daemon.py", line 192, in add_doc
    modtime=self.get_node_mtime(node),
  File "/opt/rhodecode-venv/lib/python2.6/site-packages/rhodecode/lib/indexers/daemon.py", line 158, in get_node_mtime
    return mktime(node.last_changeset.date.timetuple())
  File "/opt/rhodecode-venv/lib/python2.6/site-packages/rhodecode/lib/vcs/utils/lazy.py", line 40, in __get__
    value = self._func(obj)
  File "/opt/rhodecode-venv/lib/python2.6/site-packages/rhodecode/lib/vcs/nodes.py", line 312, in last_changeset
    return self.changeset.get_file_changeset(self.path)
  File "/opt/rhodecode-venv/lib/python2.6/site-packages/rhodecode/lib/vcs/backends/git/changeset.py", line 281, in get_file_changeset
    return self.get_file_history(path, limit=1)[0]
IndexError: list index out of range

[Marcin Kuzminski]

It looks like a bug, could you somehow share this repository for us to check out ?

Marcin Kuzminski

--
You received this message because you are subscribed to the Google Groups "rhodecode" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rhodecode+...@googlegroups.com.
To post to this group, send email to rhod...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Martin Pradny]

I have over 140 Git repositories in RhodeCode and from the stack-trace I'm not able to guess which is causing troubles. Is there some way to  isolate the problem using some debug parameters? I wasn't able to find any and I can't imagine use -index-only option to do so.

If we narrow it down I might be able to share the repo, but I can share all.

Thanks,

 Martin

[Marcin Kuzminski]

Hi,

you could get this info from logs when whoosh does indexing, it will show each repo name before going into indexing changesets and files.

You can isolate the issue with --index-only=first,second  but that could be a lot of guessing :)

Marcin Kuzminski

[Martin Pradny]

OK, I changed configuration of logging handlers and now I see the info. I also found one repo that has the issue just by trying few of them.

What's best way to share the repo with you? I can probably publish this one on github, or try to upload it to RodeCode demo site, or just send you archive from filesystem.

Btw. firefox doesn't want to let me go demo.rhodecode.org now (there is even no button to create security exception). Details bellow.

Technical Details
       
        demo.rhodecode.org uses an invalid security certificate.

The certificate is only valid for the following names:
  secure.rhodecode.org , rhodecode.org 

(Error code: ssl_error_bad_cert_domain)

[Marcin Kuzminski]

Can you try to clear your firefox cache ? or use chrome demo.rhodecode.org doesn't use ssl at all. i'm not sure why you get redirected to ssl version ?

Marcin Kuzminski

[Marcelo Bissaro]

 This Firefox issue happens to me as well. There is an issue opened against Mozilla team to fix it: https://bugzilla.mozilla.org/show_bug.cgi?id=769994
 
 And looks like it is already fixed on Firefox version 23 (the latest available for us is 22). We should see version 23 soon, I guess.
 
 Until that, I use Chrome for demo.rhodecode.org and other sites that show me the same error.

[Martin Pradny]

For some reason firefox redirects me to SSL version even when I enter non ssl address. Chrome doesn't.

Anyway, back to the issue. I tried to upload one of repositories that cause troubles to
http://demo.rhodecode.org/whoosh-crash

Crash on my system looks like:
2013-07-03 15:22:26.471 DEBUG [whoosh_indexer]     >> /home/scm/Actis/Dochazka/Forms/($$SearchTemplateDefault).form

As other repo failed on similar file my assumption is that it crashes on files with $ in name .

I both cases it is export of application from IBM Notes/Domino where such character is used to name system elements.

Hope this helps

 Martin

[marcin]

I found the reason for this, unfortunately it's not whoosh that is broken, it's our internal vcs lib that cannot properly handle special chars in filenames.

We use internal git commands to speed things up, and if we pass some filenames that have

special chars like $$ " ` etc, you have to escape it.

so instead doing git log -- /home/scm/Actis/Dochazka/Forms/($$SearchTemplateDefault).form

you have to do git log -- /home/scm/Actis/Dochazka/Forms/(\$\$SearchTemplateDefault).form

proper fix will require lot of tests. Hope to get that fixed asap.

Also a side note:

marcink@/tmp # git clone http://demo.rhodecode.org/whoosh-crash                                                                                                                                                                                      [11:38:40 on 04/07/2013]

Cloning into 'whoosh-crash'...

...

marcink@/tmp # cd whoosh-crash                                                                                                                                                                                                                       [11:38:53 on 04/07/2013]

marcink@/tmp/whoosh-crash ‹master*› # git status                                                                                                                                                                                                         [11:38:56 on 04/07/2013]

# On branch master

# Untracked files:

# "Code/Agents/Dogeneruj chybe\314\214ji\314\201ci\314\201 osoby.lsa"

# "Code/Agents/Nastaveni\314\201.fa"

# "Code/Agents/Rozes\314\214li pla\314\201n.lsa"

# "Forms/Profilovy\314\201 dokument.form"

# "Forms/Stav Zame\314\214stnance.form"

# "Forms/Z\314\214a\314\201dost o dovolenou.form"

# "Forms/Z\314\214a\314\201dost o schva\314\201leni\314\201 docha\314\201zky.form"

# "Views/(Neukonc\314\214ene\314\201 Akce dle Osoby).view"

# "Views/Akce_5cDle Me\314\214si\314\201ce.view"

# "Views/Akce_5cDle jme\314\201na.view"

# "Views/Dle jme\314\201na.view"

# "Views/Dovolena_5cKalenda\314\201r\314\214.view"

# "Views/Reports_5cDle jme\314\201na.view"

# "Views/Z\314\214a\314\201dosti_5cNove\314\201.view"

# "Views/Z\314\214a\314\201dosti_5cSchva\314\201lene\314\201.view"

# "Views/Z\314\214a\314\201dosti_5cZami\314\201tnute\314\201.view"

nothing added to commit but untracked files present

marcink@/tmp/whoosh-crash ‹master*› #

Because all other odd filenames in that repo i get immediate untracked files after a clone from that repo ?

[Martin Pradny]

Hi,

 clone works fine on Windows and since IBM Domino Designer only works on windows it's fine for us. Those escaped characters are Czech characters that we have to use, since those are quite often labels that are visible to users. I

$ git clone http://demo.rhodecode.org/whoosh-crash
Cloning into 'whoosh-crash'...
remote: Counting objects: 255, done.
remote: Compressing objects: 100% (133/133), done.
remote: Total 255 (delta 114), reused 255 (delta 114)
Receiving objects: 100% (255/255), 570.38 KiB | 605 KiB/s, done.
Resolving deltas: 100% (114/114), done.

martin.pradny@PRADNY-E6410 /e/Temp/git/whoosh/tmp
$ cd whoosh-crash/

martin.pradny@PRADNY-E6410 /e/Temp/git/whoosh/tmp/whoosh-crash (master)
$ git status
# On branch master
nothing to commit, working directory clean

I also tried it on linux and it seems fine too (but on filesystem those characters are not escaped)

[root@source git]# git clone http://demo.rhodecode.org/whoosh-crash
Cloning into 'whoosh-crash'...
remote: Counting objects: 255, done.
remote: Compressing objects: 100% (133/133), done.
remote: Total 255 (delta 114), reused 255 (delta 114)
Receiving objects: 100% (255/255), 570.39 KiB | 464 KiB/s, done.
Resolving deltas: 100% (114/114), done.
[root@source git]# cd whoosh-crash/
[root@source whoosh-crash]# git status
# On branch master
nothing to commit (working directory clean)
[root@source whoosh-crash]#

I'm no Linux or Git guru, but I think it should be possible to make both use unicode filenames.

Also Rhodecode and woosh seem to work with those repos fine. Only problem I found was with $

Next 2 weeks I'm on holiday, so no rush here. Also search is not critical feature for us. I just wanted to clean some things here (did rhodecode upgrade, ssl setup ...)

Thanks

Martin