Do I need to start hadoop nameserver in order to start an elephantdb-server?

[Tushar Deshpande]

Hi,

I am trying to start the elephantdb-server using information given by
Nathan Marz in following post: https://groups.google.com/d/msg/elephantdb-user/pj1BkR3OyUU/3lw02OguU5UJ

This post says that to start the server, I need to run the class
elephantdb.main with the following arguments:

1) Location of global-conf.clj
2) Location of local-conf.clj
3) A token

I believe that In the local config, I need to specify the location of
the hadoop name server.  E.g., :hdfs-conf {"fs.default.name" "hdfs://localhost:9000"}}

Does this mean that I need to start the hadoop name server on my
machine to use elephantdb-server?  I did not have to start the hadoop
name server to load the data into elephantdb-server.

My global config is:
{ :replication 1
  :hosts ["localhost"]
  :port 3578
  :domains {"gendercount" "/home/tushar/tmp/elephantdb/gendercount3"}
}

My local config is:
{:local-root "/tmp/elephantdb"
 :download-rate-limit 1024
 :update-interval-s 60 ;; check for domain updates every minute
 :hdfs-conf {"fs.default.name" "hdfs://localhost:9000"}
}

I would appreciate any other help about the local and global config.


Best regards,

Tushar Deshpande

[Soren Macbeth]

That tutorial is out of date. I'm going to write an updated one soon.

When you wrote out your domains from the hadoop job, where did you store them? If you stored them in a local install of hadoop on the HDFS filesystem, then yes, you have to have that local hadoop install running. If you just wrote them out to your local file system, then just add :blob-conf {"fs.default.name" "file:///"} below your :hdfs-conf key.

If you don't have hadoop running locally, you still need to include the hadoop jars on the servers classpath. The easiest way to do this is create an uberjar. From the elephantdb-server directory, run "lein with-profile provided uberjar". then you can run it like any java jar, passing the location of your global-conf and then the local-conf. the token parameter is no longer used.

--
You received this message because you are subscribed to the Google Groups "elephantdb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elephantdb-us...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
http://about.me/soren

[Tushar Deshpande]

Thanks Soren.

I wrote out my domains on my local file system. So, as you have
suggested, I would use :blob-conf {"fs.default.name" "file:///"}. My
updated local-conf looks as:

{:local-root "/tmp/elephantdb"
:download-rate-limit 1024
:update-interval-s 60 ;; check for domain updates every minute
:hdfs-conf {"fs.default.name" "hdfs://localhost:9000"}

:blob-conf {"fs.default.name" "file:///"}
}

Also, I have created the uberjar. Now, I would try starting the elephantdb
server and let you know the result.


Best regards,

Tushar Deshpande

> You received this message because you are subscribed to a topic in the
> Google Groups "elephantdb-user" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elephantdb-user/PqM2jdstD80/unsubscribe?hl=en.
> To unsubscribe from this group and all its topics, send an email to

[Tushar Deshpande]

Hi Soren,

I was able to start the elephantdb-server using following local config.

{:local-root "/tmp/elephantdblocal"

 :download-rate-limit 1024
 :update-interval-s 60 ;; check for domain updates every minute

 :blob-conf {"fs.default.name" "file:///"}
}

The server started correctly and printed following messages:
57   [clojure-agent-send-off-pool-2] 2013-04-16 13:00:37,858 INFO  jackknife.logging  - Finished opening domain at /tmp/elephantdblocal/gendercount
111  [clojure-agent-send-off-pool-0] 2013-04-16 13:00:37,912 INFO  jackknife.logging  - 1366060649636 is already loaded.

When I listed the contents of /tmp/elephantdblocal/gendercount, I saw
that it contained only two files:  1366060649636.version and
domain-spec.yaml.  I did not see any data files there.  Why did this
happen?  Would the data be copied there only after I try to
access it?  Is the location /tmp/elephantdblocal/gendercount being
used as a local cache?


Thank you,

Tushar

[Phil Kallos]

+1 a tutorial would be very helpful in getting started with elephantdb-server. I'd be happy to contribute to the setup documentation once I wrap my head around things but for the moment I am just getting started and I feel completely useless, having never used clojure or lein before.

Soren if you have a moment could you post a quick bullet-point guide for "elephantdb-server for dummies?"

* clone into https://github.com/nathanmarz/elephantdb

* cd into elephantdb-server

* set up domain-spec.yaml, local-conf.clj, global-conf.clj and place them ???

* ??? lein run <path to configs> ???

* have fun with elephantdb-server!

Something quick to get me bootstrapped would be great, I'd be happy to send a PR to add to the README once I get off the ground.

Would be much appreciated :)

Thanks,

Phil