Test databases were not set up during vagrant setup

[Jen Smith]

Hello

Just came across a strange error when trying to set up the vagrant environment. (Which is, by the way, awesome).

Following instructions here:

https://github.com/jorgej/RapidFTR/wiki/Using-a-VM-for-development

So Stefania and Linda both followed exactly the instructions this evening. Stefania's setup went fine, but Linda's for some reason failed to create the test database. This meant that we got a http connection refused style error during the rake default task.

Fixed it by RAILS_ENV=test rake couchdb:create

Any clue why this might be an issue?

Thanks

jen

[Jaco Pretorius]

+1 to the Vagrant Environment being awesome

[John D. Hume]

On Jul 4, 2012 3:03 PM, "Jen Smith" <jennifer....@gmail.com> wrote:
> Linda's for some reason failed to create the test database. This meant that we got a http connection refused style error during the rake default task.
>
> Fixed it by RAILS_ENV=test rake couchdb:create

No clue offhand. I don't suppose you still have the output from the first time through?

The most common issue is that Solr (or Jetty or Java) won't start in time or at all. I don't think I've seen some couch databases created but not others.

[Jennifer Smith]

I don't have the output sorry. I can't recall when the test databases get created the first time anyway- I don't think couchdb:create will by default (it's env related isnt it?). I would have assumed that maybe it is getting created as part of the test run. 

Will experiment on my version and delete the test databases to see if I can recreate the problem.

Jen

Sent from my iPhone

[Mark Taylor]

I am now having this issue setting up my machine. The first time I ran "rake couchdb:create db:seed app:run" it ran fine and was started and I could connect to the website from my host machine. however, if I try and run the command again, or simply rake on its own to run the tests, it now fails:

vagrant@lucid32:/vagrant$ rake --trace

(in /vagrant)

** Invoke default (first_time)

** Invoke ci:build (first_time)

** Invoke sunspot:stop (first_time)

** Execute sunspot:stop

sunspot-solr stop

** Invoke sunspot:start (first_time)

** Invoke environment (first_time)

** Execute environment

** Execute sunspot:start

sunspot-solr start -d tmp/sunspot_index -p 8983

rake aborted!

Connection refused - connect(2)

/usr/local/lib/ruby/1.8/net/http.rb:560:in `initialize'

/usr/local/lib/ruby/1.8/net/http.rb:560:in `open'

/usr/local/lib/ruby/1.8/net/http.rb:560:in `connect'

/usr/local/lib/ruby/1.8/timeout.rb:53:in `timeout'

/usr/local/lib/ruby/1.8/timeout.rb:101:in `timeout'

/usr/local/lib/ruby/1.8/net/http.rb:560:in `connect'

/usr/local/lib/ruby/1.8/net/http.rb:553:in `do_start'

/usr/local/lib/ruby/1.8/net/http.rb:542:in `start'

/usr/local/lib/ruby/1.8/net/http.rb:1035:in `__request__'

/usr/local/lib/ruby/gems/1.8/gems/rest-client-1.3.0/lib/restclient/net_http_ext.rb:17:in `request'

/usr/local/lib/ruby/1.8/net/http.rb:845:in `post'

/usr/local/lib/ruby/gems/1.8/gems/rsolr-0.12.1/lib/rsolr/connection/net_http.rb:22:in `post'

/usr/local/lib/ruby/gems/1.8/gems/rsolr-0.12.1/lib/rsolr/connection/requestable.rb:33:in `request'

/usr/local/lib/ruby/gems/1.8/gems/rsolr-0.12.1/lib/rsolr/client.rb:34:in `request'

/usr/local/lib/ruby/gems/1.8/gems/rsolr-0.12.1/lib/rsolr/client.rb:22:in `update'

/usr/local/lib/ruby/gems/1.8/gems/rsolr-0.12.1/lib/rsolr/client.rb:76:in `delete_by_query'

/usr/local/lib/ruby/gems/1.8/gems/sunspot-1.1.0/lib/sunspot/indexer.rb:55:in `remove_all'

/usr/local/lib/ruby/gems/1.8/gems/sunspot-1.1.0/lib/sunspot/session.rb:173:in `remove_all'

/usr/local/lib/ruby/gems/1.8/gems/sunspot-1.1.0/lib/sunspot/session.rb:173:in `each'

/usr/local/lib/ruby/gems/1.8/gems/sunspot-1.1.0/lib/sunspot/session.rb:173:in `remove_all'

/usr/local/lib/ruby/gems/1.8/gems/sunspot-1.1.0/lib/sunspot.rb:414:in `remove_all'

/vagrant/app/models/searchable.rb:51:in `reindex!'

/vagrant/lib/tasks/sunspot.rake:9

/usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:636:in `call'

/usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:636:in `execute'

/usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:631:in `each'

/usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:631:in `execute'

/usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:597:in `invoke_with_call_chain'

/usr/local/lib/ruby/1.8/monitor.rb:242:in `synchronize'

/usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:590:in `invoke_with_call_chain'

/usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:607:in `invoke_prerequisites'

/usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:604:in `each'

/usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:604:in `invoke_prerequisites'

/usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:596:in `invoke_with_call_chain'

/usr/local/lib/ruby/1.8/monitor.rb:242:in `synchronize'

/usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:590:in `invoke_with_call_chain'

/usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:607:in `invoke_prerequisites'

/usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:604:in `each'

/usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:604:in `invoke_prerequisites'

/usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:596:in `invoke_with_call_chain'

/usr/local/lib/ruby/1.8/monitor.rb:242:in `synchronize'

/usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:590:in `invoke_with_call_chain'

/usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:583:in `invoke'

/usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2051:in `invoke_task'

/usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2029:in `top_level'

/usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2029:in `each'

/usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2029:in `top_level'

/usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2068:in `standard_exception_handling'

/usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2023:in `top_level'

/usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2001:in `run'

/usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2068:in `standard_exception_handling'

/usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:1998:in `run'

/usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/bin/rake:31

/usr/local/bin/rake:19:in `load'

/usr/local/bin/rake:19

I've tried killing the java process for sunspot and it makes no difference :-(

[Mark Taylor]

Well...I have it working now, but only by trying to run rake and kill the sunspot process a few times...not sure how reliable it will be for me going forwards...

[John D. Hume]

This has been an issue for many people using the VM and (I think) some running natively. The sunspot restart has a hard-coded wait to give Solr time to come up, but it's often insufficient. At other times it seems Solr never comes up.

If anyone can dig into this it would be much appreciated. The easy next step is to have rake check to see when Solr is responding and only then proceed to reindex children. The messier thing is digging into why sometimes it doesn't come up at all.

-- typed with my thumbs

[Jorge Just]

just bumping john's post in case anyone is interest in tackling this.

jorge

[Mark Taylor]

I had a look at this at the London CodeJam tonight, but I still haven't resolved it. I had a bried period where I thought allocating more memory to the VM had fixed it (inspired by reading this: http://comments.gmane.org/gmane.comp.jakarta.lucene.solr.user/11070), but it's since recurred.

From running the start command manually it appears that it just sometimes takes a long time for it to be up and start listening on port 8983; I came across an article here: https://issues.apache.org/jira/browse/SOLR-2019 about Jetty taking a long time to start (and I think solr uses jetty?), which was fixed a while ago, but I don't know how old our version is as when I run sunspot-solr -v it says "unknown version".

So not much progress really, I'm afraid :-(

Mark

On Wednesday, 25 July 2012 01:53:32 UTC+1, jorge wrote:

just bumping john's post in case anyone is interest in tackling this.

jorge

On Wed, Jul 18, 2012 at 7:10 PM, John D. Hume <> wrote:

This has been an issue for many people using the VM and (I think) some running natively. The sunspot restart has a hard-coded wait to give Solr time to come up, but it's often insufficient. At other times it seems Solr never comes up.

If anyone can dig into this it would be much appreciated. The easy next step is to have rake check to see when Solr is responding and only then proceed to reindex children. The messier thing is digging into why sometimes it doesn't come up at all.

-- typed with my thumbs

On 4 Jul 2012, at 23:15, "John D. Hume" <> wrote:

[Jorge Just]

Thanks, Mark

I'm bumping this in case anyone has any ideas.

thanks

jorge

[Jaco Pretorius]

Hi

Is there a reliable way to reproduce this problem?

Jaco

[Mark Taylor]

Not sure, all I did was install the VM by following the instructions on the RapidFTR wiki.

[Johannes Thoenes]

Hello,

I dug into this and found a solution. You can "fix" the problem by open the file on the vagrant box /usr/lib/jvm/java-6-openjdk/jre/lib/security/java.security, find and change the following line

securerandom.source=file:/dev/urandom

by adding a dot "."

securerandom.source=file:/dev/./urandom

For those who are interested - solution part ends here ;-): What was the problem?

Jetty is using Java's SecureRandom.getInstance("SHA1PRNG") to create secure random session ids[1]. In Java 1.4 this would have used /dev/urandom when specified. However, /dev/urandom will not block and immidiately return, even if not enough entropy (from IO, networking etc.) is available. This is why starting with Java 5, /dev/random is used even if /dev/urandom is specified. And /dev/random will block the process, until enough entropy is available. This explains, why the behaviour was so strange - it depended on the things you did on the VM. 

Another work-around is to use /dev/./urandom as RNG[2], which you can either do in the java.security file for every java application on a linux box or by using the -Djava.security.egd=file:/dev/./urandom option on the java command. Because java internally uses the string "/dev/random" pointing to the same source with a different string, will work. 

Hopes this makes some people more productive.

Best regards

Johannes

[1] http://docs.codehaus.org/display/JETTY/Connectors+slow+to+startup

[2] http://bugs.sun.com/view_bug.do;jsessionid=ff625daf459fdffffffffcd54f1c775299e0?bug_id=6202721

[John D. Hume]

On Wed, Sep 19, 2012 at 2:51 PM, Johannes Thoenes
<jtho...@thoughtworks.com> wrote:
> I dug into this and found a solution. You can "fix" the problem by open the
> file on the vagrant box
> /usr/lib/jvm/java-6-openjdk/jre/lib/security/java.security, find and change
> the following line
>
>> securerandom.source=file:/dev/urandom
>
>
> by adding a dot "."
>
>> securerandom.source=file:/dev/./urandom

Awesome research! I think this wins the Mines of Moria prize* (because
you delved so deep).
http://images.wikia.com/lotr/images/8/89/Theminesofmoriaset.jpg

* no actual prize

--
http://elhumidor.blogspot.com/

[Johannes Thoenes]

Thanks ;-)