JAQL Type Evaluation + Second-Order Functions

[Rob Stewart]

Hi,

I have a few questions about JAQL, once again.

I'm interested in the page: http://code.google.com/p/jaql/wiki/PlanLanguage , and I'm particularly interested in the sentence "The mapReduce() function touches on a interesting feature of Jaql, namely that Jaql is a second-order language".

Does that imply that JAQL is a lazily evaluated language? That is, where JAQL, with potentially erroneous data structures, may be computed successfully as a MapReduce job if those structures are never instantiated from other sections of the JAQL program? (like Haskell, for instance). 

I am always slightly unsure if my definition of "second-order" and "higher-order" are the same. For instance the page: http://www.ibm.com/developerworks/web/library/wa-dojohadoop1/ states in the conclusion that Pig and Hive are higher-order "frameworks", though I'm not sure that this implies "higher order functions" (maybe with Pig). I am also not entirely clear why JAQL is defined as a second-order language in the link above, and not a higher-order language. 

And to clarify, JAQL can be described as a purely functional programming language?

Finally - What are the rules with data types in JAQL? Are all variables explicitly typed by the programmer, or are there mechanics in the JAQL compiler to implicitly deduce data types?

thanks,

Rob Stewart

[vuk.ercegovac]

The use of the phrase "second-order" is an error-- it should read
"higher-order".
Thats been corrected-- thanks for highlighting this. Jaql's type
system includes
jaql functions, jaql functions can have function arguments, and
functions can be
returned from functions. I do not believe that Pig or Hive have these
features.

Jaql does include lazy evaluation. Dynamic, runtime errors are
possible due to type
mismatches. Jaql's design borrows a lot from functional languages, but
I would not call
it "pure".

When it comes to data types, Jaql gets type information dynamically at
runtime and can be
told about the types of variables through a schema language that we've
developed for json
(borrows from the existing work on xml schema and json schema). You
can see examples of
schema in the various unit tests: src/test/com/ibm/jaql/
schemaQueries.txt. You can explicitly use schame to validate data to
schema using "check" and schema is used in various readers, for
example "del" format. The plan is to increase the use of schema
throughout Jaql. Internally, we already do schema inference-- so with
whatever schema information we do get, either explicitly or implicitly
via expression schemas, we do infer schemas. One place where we make
use of schema information is between map and reduce. During rewrite,
after map-reduce jobs have been obtained, we use schema inference to
determine the schema of map outputs. This is used to serialize the
data more efficiently, thereby reducing the amount of data needed to
be shipped over the network. Look at the JaqlTemp file type, for
example, to see how we connect schema and serialization/
deserialization.

On Aug 30, 10:39 am, Rob Stewart <robstewar...@googlemail.com> wrote:
> Hi,
>
> I have a few questions about JAQL, once again.
>
> I'm interested in the page:http://code.google.com/p/jaql/wiki/PlanLanguage,
> and I'm particularly interested in the sentence "The mapReduce() function
> touches on a interesting feature of Jaql, namely that Jaql is a second-order
> language".
>
> Does that imply that JAQL is a lazily evaluated language? That is, where
> JAQL, with potentially erroneous data structures, may be computed
> successfully as a MapReduce job if those structures are never instantiated
> from other sections of the JAQL program? (like Haskell, for instance).
>
> I am always slightly unsure if my definition of "second-order" and

> "higher-order" are the same. For instance the page:http://www.ibm.com/developerworks/web/library/wa-dojohadoop1/states in the