how to get started in Genie3

[ric....@liulishuo.com]

hi, 

we are trying to use Genie as a Job Execution Service to our EMR clusters, I reference the wiki at _installation section, but I don't think that is very clear to us  such as: 

• what are the next steps after `java -jar genie.jar` ?
• how to register our EMR cluster as a Clusters Resource in Genie, we have already try the `*.yml` in ~/.genie/，but after placed the configuration file, we still can not view the clusters information on UI even though the Genie restarted 
• should we manually register the Applications / Commands / Clusters to Genie by python script ?  like the ./init-demo.py in Demo.  

 What is the best practice for Genie?  someone can give more steps?  it would be appreciated.

thanks.  

 

 

 

[Marco Primi]

Hello, 

On Nov 28, 2017, at 11:27 PM, ric....@liulishuo.com wrote:

hi, 

we are trying to use Genie as a Job Execution Service to our EMR clusters, I reference the wiki at _installation section, but I don't think that is very clear to us  such as: 

• what are the next steps after `java -jar genie.jar` ?

This brings up a service with a REST API, the next thing you want to do is probably use it to register your commands/clusters/applications so that your users can use the jobs API to submit MR tasks.

There are quite a few things you can customize, as you can see in the guide. For example, set up a persistent database. This makes sure your configuration survives server restarts and allows multiple Genie servers to be up (for fault tolerance and to accept more work than a single machine can take).

• how to register our EMR cluster as a Clusters Resource in Genie, we have already try the `*.yml` in ~/.genie/，but after placed the configuration file, we still can not view the clusters information on UI even though the Genie restarted 

• should we manually register the Applications / Commands / Clusters to Genie by python script ?  like the ./init-demo.py in Demo.  

~/.genie/application-*.yml and similar files are Spring Boot profile files. They contain properties that configure Genie itself (say, the database connection, load balancing strategy, security, etc.).

Registration of commands/clusters/applications happens via REST API.

So yes, look at init-demo.py for an example (should be a good starting point, as it does register and use a tiny EMR cluster!)

•  What is the best practice for Genie?  someone can give more steps?  it would be appreciated.

I’d recommend reviewing what the demo script do, and maybe use them as base to create and register your own entities.

For more complex examples, you can how we use Genie at Netflix in terms of entities registered and in terms of deployment.

thanks.  

 

 

 

--
You received this message because you are subscribed to the Google Groups "genie" group.
To unsubscribe from this group and stop receiving emails from it, send an email to genieoss+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sourabh rajput]

Hi

I am having same issue which you have addressed above. Since I am using own server for my Spark Job deployment through Genie, but I am unable register all  resources (clusters, commands, applications) with Genie before attempting to link them together. I already have a cluster of Spark. I have installed Genie 4.0.0-rc.2 using its jar file(Standalone). 

I am also unable to find "~/.genie/application-*.yml" file in my server.

Hoping to hear you soon.

[Marco Primi]

Hello Sourabh,

I am using own server for my Spark Job deployment through Genie

Can you elaborate on what this means? I’m not clear.

I am unable register all  resources (clusters, commands, applications) with Genie before attempting to link them together.

Are you getting API errors? Could you provide the error message and/or the genie server logs?

I am also unable to find "~/.genie/application-*.yml" file in my server.

This file is not supposed to exist unless you create it to customize Genie runtime options.

[Sourabh rajput]

Thanks for the reply.

Let me elaborate the situtation.

1. Can you elaborate on what this means? I’m not clear.

==> Since in genie refrence doc, Amazon AWS(AWR) has been used for deployment of spark job, but in my case I am using a RedHat server which itself has enough space.

2. Are you getting API errors? Could you provide the error message and/or the genie server logs?

==> I am using Postman(HTTP) for registering clusters and command via json. After registering clusters and command I register for job in which it is linked to cluster and command. But after linking the job I get the status of job as failed.

I have enclosed some attachments which specify the clusters(screenshot 140), command(screenshot 141) and jobs (screenshot 142) information respectively.

[Marco Primi]

According to the job status. The job was kicked off correctly, but immediately failed.

There are probably relevant error messages in the job outputs.

To see them, click on the “output folder” icon in the jobs view, and see ’stdout’ and ’stderr’.

You could also check the Genie server logs.

<Screenshot (140).png><Screenshot (141).png><Screenshot (142).png>

[Sourabh rajput]

Hi ,

There is no error message in the stdout (Screenshot (145).png) and strerr (Screenshot (146).png) folder and in the genie log(Screenshot (148).png) there is a message command not found.

[Marco Primi]

During launch, Genie generates a ‘run’ script for your job that executes your dependencies’ setup script, if any.

It seems that there’s a syntax error in one of these scripts.

Try to re-run ’run’ and see if you can debug it.

Or attach it and we’ll take a look.

<Screenshot (145).png><Screenshot (146).png><Screenshot (147).png><Screenshot (148).png>