What is the real response of Livy Spark REST server GET /batches method

[luis.fern...@gmail.com]

I"m trying to use the Livy server as a REST endpoint for my Spark batch jobs. I"m testing it with the standard sparkpi example

curl -i -XPOST -H 'Content-Type: application/json' localhost:8998/batches -d '{"className":"org.apache.spark.examples.SparkPi", "file": "/usr/local/Cellar/apache-spark/1.4.0/libexec/lib/spark-examples-1.4.0-hadoop2.6.0.jar", "args":"100"}';

Which works fine, but when it request the output with 

curl -i -XGET -H 'Content-Type: application/json' localhost:8998/batches/0;

I get this instead of of what's in the documentation. I don't see the lines property or the output of the job in the log file. Where do I find the output? Thank you; 

{"id":0,"state":"success","log":["15/08/28 15:07:20 INFO Utils: path = /private/var/folders/qz/4sr7vgmx0w3_m0hgz4836bz40000gq/T/spark-6ea292b2-c359-42f0-a33d-22746efc9938/blockmgr-58fd1023-43cb-4f9a-9f87-8c5b8088e06a, already present as root for deletion.","15/08/28 15:07:20 INFO MemoryStore: MemoryStore cleared","15/08/28 15:07:20 INFO BlockManager: BlockManager stopped","15/08/28 15:07:20 INFO BlockManagerMaster: BlockManagerMaster stopped","15/08/28 15:07:20 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!","15/08/28 15:07:20 INFO SparkContext: Successfully stopped SparkContext","15/08/28 15:07:20 INFO Utils: Shutdown hook called","15/08/28 15:07:20 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.","15/08/28 15:07:20 INFO Utils: Deleting directory /private/var/folders/qz/4sr7vgmx0w3_m0hgz4836bz40000gq/T/spark-6ea292b2-c359-42f0-a33d-22746efc9938","15/08/28 15:07:20 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports."]}%

Response Body

name	description	type
id	batch list	list
state	The state of the batch	batch state
lines	The output of the batch job	list of strings

[Erick Tryzelaar]

Hello Luis,

Oops, the documentation is a little out of date. I'll update it, but in the meantime, you should be able to do:

% curl -i -XGET -H 'Content-Type: application/json' localhost:8998/batches/0/log?size=$SIZE&from=$FROM

Where you can use $FROM and $SIZE to page through the log lines.

Let me know if you have trouble with it.

-Erick

To unsubscribe from this group and stop receiving emails from it, send an email to hue-user+u...@cloudera.org.

[luis.fern...@gmail.com]

It works perfectly. Thank you.