Re: Running Sparrow

[Kay Ousterhout]

Hi Kiran,

Adding the Sparrow user group list here, since this question may be helpful to others!

If you want to run sleep tasks, you'll probably want to write your own frontend and backend, which will be similar to the proto frontend/backend (except that instead of running a CPU-intensive computation for some period of time, they'll just sleep).

The use of the backend is to actually run the tasks.  An application running with Sparrow has 3 components.  There is a Sparrow Daemon running on each machine; each Sparrow Daemon includes a Scheduler (which schedules tasks) and a NodeMonitor, which monitors what's running on the machine, ensures that some maximum number of tasks are running, and communicates with other schedulers that are trying to run tasks.  Now, applications are decomposed into a frontend and a backend.  The frontend submits jobs to the Sparrow scheduler.  A backend runs on each worker machine; when a Sparrow daemon on a worker machine is assigned a task to run on the machine, it passes the task description (which is opaque to Sparrow) to the worker.  You can see this if you look at the ProtoBackend code -- the ProtoBackend implements the launchTask method (part of the BackendService interface), which the Sparrow node monitor uses to pass tasks to the backend.  To run sleep tasks, you'll want to replace the code in runBenchmark in the ProtoBackend to sleep for a period of time (most of the other code in ProtoBackend is for logging purposes -- it records how many tasks were launched and the rate of task launch for debugging purposes).

You're exactly right that, without a configuration file, the frontend will use a default configuration.

I'm guessing the problem that you're having has to do with inconsistency in how the hostname is represented.  Can you send the full log messages?

-Kay

On Sun, Oct 27, 2013 at 5:46 PM, Kiran Ramamurthy <kram...@hawk.iit.edu> wrote:

Hi Kay,

Sorry I forgot to mention the tasks I need to run. For my benchmarking purpose I need to run sleep tasks.

---------- Forwarded message ----------
From: Kiran Ramamurthy <kram...@hawk.iit.edu>
Date: Sun, Oct 27, 2013 at 11:19 AM
Subject: Re: Running Sparrow
To: Kay Ousterhout <kayous...@gmail.com>

Hi Kay,

I am following the readme document and following are the steps I followed,

1. Changed the deploy/example_sparrow_configuration.conf to add my IP address. Here since I am running the localhost I have given 127.0.1.1 retaining the port number specified in the document

2. I start sparrow with the following command

java -XX:+UseConcMarkSweepGC -cp target/sparrow-1.0-SNAPSHOT.jar edu.berkeley.sparrow.daemon.SparrowDaemon -c deploy/example_sparrow_configuration.conf

3. I start sparrow back-end with the following command

java -cp target/sparrow-1.0-SNAPSHOT.jar edu.berkeley.sparrow.prototype.ProtoBackend

4. I start sparrow front-end with the following command

java -cp target/sparrow-1.0-SNAPSHOT.jar edu.berkeley.sparrow.prototype.ProtoFrontend

With my knowledge, running front-end without a configuration file specified, has its own configurations and it will run tasks by default. Am I on the right track here? I see many threads being created to send tasks to the scheduler.

If so, what is the use of back-end? Does it need to be running if the front-end is running?

I am also facing the following error and the scheduler halts,

DEBUG edu.berkeley.sparrow.daemon.nodemonitor.TaskLauncherService  - Didn't receive a task for request 127.0.1.1_3099

ERROR edu.berkeley.sparrow.daemon.scheduler.UnconstrainedTaskPlacer  - Node monitor THostPort(host:kiran-Dell-System-XPS-L502X, port:20502) not in list of outstanding reservations

Readme says, we can provide a conf file for front-end specifying '-c'. I am not sure of the contents of the conf file and what should go in it.

Sorry for the big mail. Please help me set it up.

Thanks,

Kiran

On Sun, Oct 27, 2013 at 12:22 AM, Kay Ousterhout <kayous...@gmail.com> wrote:

Hi Kiran,

Happy to help get this setup!  What type of job are you trying to submit?  The Sparrow code base has a frontend that can be used to submit basic jobs that use the CPU for a set amount of time (the ProtoFrontend/Backend).  If you'd like to run more complex queries, you'll need to use Spark, which is more work to setup.  Let me know which of these applies to you!

-Kay

On Sat, Oct 26, 2013 at 12:10 PM, Patrick Wendell <pwen...@gmail.com> wrote:

Hi there! Cc'd is Kay who is currently leading the project. She can help you get things up and running.

On Fri, Oct 25, 2013 at 9:58 AM, Kiran Ramamurthy <kram...@hawk.iit.edu> wrote:

Hi,

I am Kiran, student at Illinois Tech, Chicago. I am benchmarking certain systems which includes Sparrow. I was able to download and deploy.

But I am not getting to know how to write the conf file to submit jobs. Can you please direct me if there is any detailed description on how to setup and run.

Any help regarding this would be deeply appreciated.

Thanks,

--

Kiran Ramamurthy

Illinois Institute of Technology

Data-Intensive Distributed Systems Laboratory

--

Kiran Ramamurthy

Illinois Institute of Technology

Data-Intensive Distributed Systems Laboratory

--

Kiran Ramamurthy

Illinois Institute of Technology

Data-Intensive Distributed Systems Laboratory

[dhirendra singh]

Hi Kay,

Can you please help me in fixing the below problem:

1. I have deployed the system and have ru the below three commands to invoke node monitor , ProtoBackend and ProtoFrontend and as soon ProtoBackend is invoked, it says: Client is registered and when I run the below command for ProtoFronend, everything starts executing but after everything is stopped. I had to kill all the running "java" processs manually to stop all things completely. Am I doing it correct?

2. I am new to the computing, so facing some problem in understanding the whole code of Sparrow. Can you please help , how can I check how many tasks are being submitted and how much time it takes to complete the whole tasks. I ran the system on two nodes and its running fine but don't know how to compute the performance. I wrote a script which kills all the processes of "java" once everything stops to execute but then I have to monitor the system when the execution is done and then run that script to kill the jobs which gives me the start and end date. Am I doing it right?

it would be really helpful if you could help me on this.

Thanks,

Dhirendra