Getting started with JEQL...

[Sunburned Surveyor]

I've got a little project at work that I want to try JEQL on. It's a
simple GUI tool that accesses owner and parcel information for a
municipality given an address or lattitude/longitude. The client
doesn't want an enterprise GIS and I can't get support for OpenJUMP at
this point.

I downloaded the JEQL distro and checked out the examples. I must
admit that I am a little confused about how to even get started with
JEQL. I wanted to ask some questions about statements in the example
script. If this hand holding isn't possible then I will wait until a
user guide comes out. :] If it is possible then I would be willing to
put together some quick "Getting started with JEQL" docs.

Here are my questions:

Script Statement: baseDir = "C:\data\";
Question: Does this create a variable called baseDir that points to a
String? Or does it point to a Folder or File? It looks like JEQL is
not statically typed. Is this correct?

Script Statement: CSVReader tCSV file: inFile;
Question: I think we are creating a reader for a CSV file named tCSV
here. Is that correct? If this is the case what does the "file:
inFile;" part mean? (I know we are telling the CSVReader to use the
CSV file pointed to by the inFile variable, but I don't understand the
abstract syntax of this part. Are these like arguments to the
CSVReader constructor?)

Script Statement: tCSVData = select * from tCSV offset 1;
Question: In this statement are we selecting all of the "cells" in the
CSV except for the first line? If this is correct, could you use an
offset greater than 1? How about selecting specific rows in a CSV file
by their position in the file, or a range of rows? How would you do
this?

Script Statement: tData = select
Value.toDouble(col0) as lat,
Value.toDouble(col1) as lon,
col2 as name,
col3 as address,
col4 as city,
col5 as provState,
col6 as postalZip,
col7 as country,
col8 as phone
from tCSVData;
Question: It seems like tData is a "row". Is that correct? Are we
creating tData for a single row in the CSV file, or for all rows?
There is no loop structure apparent so I wasn't sure about this. Are
"co10" and "co11" column identifiers? If this is correct, why are we
not specifiying column 1 and column 2, which appear to hold the
latitude and longitude values in the CSV table? Are "lat" and "lon"
datatypes? Why do we specify datatypes here, but not for baseDir,
inFile, tCSV, or tData?

I think that will get me started.

I hope I am not a major pain in the neck. If I can get some basic JEQL
syntax figured out I would like to provide helpful feedback that can
improve the language. As someone who avoids JDBC and relational
databases like the plague I am really excited about JEQL.

The Sunburned Surveyor

[Martin Davis]

Good stuff, Landon - I knew you'd be a great beta tester!

Sorry you're having a tough time getting started.  I think once you grok the JEQL paradigm you'll have a much easier time of it.

I will put together a list of answers to your questions ASAP - hopefully by the end of the day.

Martin

[Martin Davis]

On Tue, Mar 11, 2008 at 12:16 PM, Sunburned Surveyor <sunburned...@gmail.com> wrote:

I've got a little project at work that I want to try JEQL on. It's a
simple GUI tool that accesses owner and parcel information for a
municipality given an address or lattitude/longitude. The client
doesn't want an enterprise GIS and I can't get support for OpenJUMP at
this point.

 

Can you give me a more detailed description of your use case?  Then I might be able to offer some more targetted advice.

I downloaded the JEQL distro and checked out the examples. I must
admit that I am a little confused about how to even get started with
JEQL. I wanted to ask some questions about statements in the example
script. If this hand holding isn't possible then I will wait until a
user guide comes out. :] If it is possible then I would be willing to
put together some quick "Getting started with JEQL" docs.

Here are my questions:

Script Statement: baseDir = "C:\data\";
Question: Does this create a variable called baseDir that points to a
String? Or does it point to a Folder or File?  It looks like JEQL is
not statically typed. Is this correct?

 

baseDir is a variable which is being initialized to a string.  Variables in JEQL are untyped - their type is the type of whatever value they are holding.

Script Statement: CSVReader tCSV file: inFile;
Question: I think we are creating a reader for a CSV file named tCSV
here. Is that correct?

No, this statement reads a table from the file specified by the filename in inFile into a table which is stored in the variable tCSV.  Tables are first-class objects in JEQL, and can be stored in variables.
 

If this is the case what does the "file:
inFile;" part mean? (I know we are telling the CSVReader to use the
CSV file pointed to by the inFile variable, but I don't understand the
abstract syntax of this part. Are these like arguments to the
CSVReader constructor?)

This is a statement,  it's not a constructor.   There is no notion of classes in JEQL - it's simpler than that (deliberately)

Script Statement: tCSVData = select * from tCSV offset 1;

This stmt creates a new table which consists of all rows in the tCSV table except the first row.

Question: In this statement are we selecting all of the "cells" in the
CSV except for the first line? If this is correct, could you use an
offset greater than 1?

Yes.
 

How about selecting specific rows in a CSV file
by their position in the file, or a range of rows? How would you do
this?

select * from tCSV offset n limit m   (common SQL syntax, for some flavours of SQL)
 

Script Statement: tData = select
       Value.toDouble(col0) as lat,
       Value.toDouble(col1) as lon,
       col2 as name,
       col3 as address,
       col4 as city,
       col5 as provState,
       col6 as postalZip,
       col7 as country,
       col8 as phone
       from tCSVData;
Question: It seems like tData is a "row". Is that correct?

No, tData is a table
 

Are we
creating tData for a single row in the CSV file, or for all rows?

All rows - just like a select stmt normally works

There is no loop structure apparent so I wasn't sure about this. Are
"co10" and "co11" column identifiers?

Yes - they are the names of the columns in tCSV.  Add the stmt

Print tCSV limit: 1;

and you can see this

 

If this is correct, why are we
not specifiying column 1 and column 2, which appear to hold the
latitude and longitude values in the CSV table?

Currently the CSVReader doesn't use the "column names" specified in the CSV file first line (this is planned to be added).  Instead, it creates a table with fixed column names col1, col2, col3, etc.  These columns all have type String.  If you want them to have some other type, like Double, you have to use a conversion function.
 

Are "lat" and "lon"
datatypes? Why do we specify datatypes here, but not for baseDir,
inFile, tCSV, or tData?

lat and lon are converted to Doubles, but the other columns are left as strings (but are renamed)

I think that will get me started.

I hope I am not a major pain in the neck. If I can get some basic JEQL
syntax figured out I would like to provide helpful feedback that can
improve the language. As someone who avoids JDBC and relational
databases like the plague I am really excited about JEQL.

Cool - look forward to your feedback

Martin

[Sunburned Surveyor]

Thanks for the info Martin. Let me chew on it and I will get back to
you. I may have a couple more questions, but I think I am getting
closer to being able to write my first simple JEQL script.

Landon

[Sunburned Surveyor]

I'm finally getting back to this.

Martin wrote: "Can you give me a more detailed description of your use

case? Then I might be able to offer some more targetted advice."

Sure thing.

I'm working with a very small municipality. They can't afford ESRI GIS
at this point, but they will be moving that way at some point in the
future. In the meantime they want some simple "GIS" tools that they
can use in their office. Here are some example use cases:

Mr. Jones comes into the office and asks for a copy of his deed. He
gives his deed to the administrative assistant. She types the address
into a tool which pulls up a PDF copy of the grant deed.

The municipality is considering a tax assessment increase. They would
like a list of all commerical properties in the municipailty over 5
acres in size.

There is no support at this point for using OpenJUMP at the
municipality office. Even if there was, I think I would have some user
interface and reliability issues that I would like to resolve first.
As a result, I need to build some simple GUI tools that execute
predetermined queires. These could be spatial queries, non-spaital
queries, or a combination of both. I will be storing all of the data
for the query tools on a MS Windows file system. My only allowed file
formats are ESRI Shapefiles, DXF files, and CSV files. (My company
will be responsible for providing the municipality with update data
files for the query tools.)

I was going to write some basic CSV access and query tools from
scratch when I found out about JEQL. It seems like exactly what I
need.

I'm still not sure if I will use JEQL to access the ESRI Shapefiles
for the spatial queries, or if I will be using components of the
OpenJUMP API, with which I am most familiar.

The first task I would like to accomplish with JEQL is very simple. I
want to select a set of parcels from a CSV file based on a simple
attribute value, like parcel area or land use type.

Then I will attempt something spatial, like selecting all parcels
within a buffer distance of a latitude/longitude location.

All of your other answers to my questions made sense (at this point).
Except for one:

You mentioned that JEQL is dynamically typed. If this is the case,
what is the purpose of the token "CSVReader" in the script statement:
"CSVReader tCSV file: inFile;"

Also, you wrote that "Tables are first-class objects in JEQL". Is
there a list of basic operations that can be performed on tables?

Thanks for all of the help.

Landon

On Tue, Mar 11, 2008 at 2:31 PM, Sunburned Surveyor

[Martin Davis]

Thanks, Landon - good info.

There's a lot to digest in your note, so I'll start with the Jeql questions first.

The reason for the CSVReader command is that it describes the action you want to perform - i.e. reading a CSV file into a table.  This doesn't really have anything to do with dynamic typing - or perhaps I misunderstood the question?

As for operations on tables, the most obvious one is the select operator.  And then all the Readers and Writers (I'm working on getting a list of those).  There's also some functions which produce tables, such as the Generate functions.  There's a rough preliminary list of functions here:  http://tsusiatsoftware.net/jeql/ref/func.html

Now for your use cases:
- querying CSV files is a perfect case
- the spatial queries should be easy too (in fact you could join the shapefiles and the CSV files - although if they are large performance might be an issue)
- Embedding JEQL behind a GUI is probably not a bad idea.  Ultimately it is intended to be easily callable from Java, although right now I'm focussed on getting the scripting language up and running.  A JDBC-style interface might be a nifty thing to have - do you think?

Other ideas:
- would it be of interest to be able to convert your municipal data to KML, either for staff or people to be able to see properties in GE/GM?  You can do this with the KMLWriter (and possibly some reprojection to LatLong - support for this is a bit primitive right now, but if you need something here maybe we can work on building it in)
- Do you need to be able to read DXF?  I think OJ has DXF I/O, right?  That code should be easy to interface to Jeql, as an external import library.

M

[Sunburned Surveyor]

Martin wrote: "The reason for the CSVReader command is that it

describes the action you want to perform - i.e. reading a CSV file
into a table. This doesn't really have anything to do with dynamic
typing - or perhaps I misunderstood the question?"

Now I understand. The first token describes the action to perform on
the file. The second token creates a variable in which the result of
the action is stored, and the last wo (2) token describe the parameter
for the action, a file. If I have this correct the generic syntax for
the statement would look something like this:

[Action Type] [Action Result Variable] [Action Parameter]

Please correct me if I am still mixed up on this statement.

Martin wrote: "As for operations on tables, the most obvious one is

the select operator. And then all the Readers and Writers (I'm
working on getting a list of those). There's also some functions
which produce tables, such as the Generate functions. There's a rough
preliminary list of functions here:
http://tsusiatsoftware.net/jeql/ref/func.html"

Roger that.

Martin wrote: "the spatial queries should be easy too (in fact you

could join the shapefiles and the CSV files - although if they are
large performance might be an issue)"

In my case I will want to keep some of the data in separate files. I
don't think this will be a problem. It seems like JEQL handle multiple
files/tables.

Martin wrote: "Embedding JEQL behind a GUI is probably not a bad idea.

Ultimately it is intended to be easily callable from Java, although
right now I'm focussed on getting the scripting language up and
running. A JDBC-style interface might be a nifty thing to have - do
you think?"

I think JEQL probably has what I need already. The user will click a
button on the GUI, I'll execute the JEQL Query, and then I will pass
the results back to the user. I don't know that we should bog JEQL
down with a lot of GUI connectiveness. I think it will fufill its role
(as a query engine) just fine.

That doesn't mean that there might not be room for some utility code
that makes integration of JEQL into a GUI easier. For example, you
might have a wrapper that returns the results of a JEQL query
displayed in a simple JTable, or if the results or spatial, in a
LayerViewPanel.

Martin wrote: "would it be of interest to be able to convert your

municipal data to KML, either for staff or people to be able to see
properties in GE/GM? You can do this with the KMLWriter (and possibly
some reprojection to LatLong - support for this is a bit primitive
right now, but if you need something here maybe we can work on
building it in)"

This would be cool. I'll have to think about how it fits into the picture.

Martin wrote: "Do you need to be able to read DXF? I think OJ has DXF

I/O, right? That code should be easy to interface to Jeql, as an
external import library."

I can export Shapefiles from AutoCAD Map, but it isn't the most
elegant process. I think I am going to go that route for now. At some
point I need to write my own low-level parser for DXF files. 98% if my
spatial data is available in this format. If I get around to this I
will make the low-level parser available for OJ. Maybe then we can
talk about integrating a DXF reader/writer for JEQL. I'm currently
waiting for an oppportunity to write this parser on the clock, instead
of in my free time. So i'm not sure when I will get to it.

Thanks for all of your help. Now that I have an idea of how to get
started I need to set up some test data files and try some simple
queires. I'll have more questions then. Hopefully, I can get to this
in a few weeks.

Landon

[Martin Davis]

Landon,

some answers...

On Mon, Mar 24, 2008 at 4:06 PM, Sunburned Surveyor <sunburned...@gmail.com> wrote

Now I understand. The first token describes the action to perform on
the file. The second token creates a variable in which the result of
the action is stored, and the last wo (2) token describe the parameter
for the action, a file. If I have this correct the generic syntax for
the statement would look something like this:

[Action Type]  [Action Result Variable] [Action Parameter]

Please correct me if I am still mixed up on this statement.

Almost...  The syntax is:

<commmand-name> [ <default-parameter-expression> ] { <param-name>: <expression> }

(where [ ] means "optional" and { } means "0 or more" )

(I like your term "action" tho - but too late to change now!)

Commands *may* take a default parameter, for ease of typing.   In general they take 0 or more named parameters, which are single expressions.  Usual default parameters are also available as named params too.

In my case I will want to keep some of the data in separate files. I
don't think this will be a problem. It seems like JEQL handle multiple
files/tables.

Yup

I think JEQL probably has what I need already. The user will click a
button on the GUI, I'll execute the JEQL Query, and then I will pass
the results back to the user. I don't know that we should bog JEQL
down with a lot of GUI connectiveness. I think it will fufill its role
(as a query engine) just fine.

Yes, the query engine should definitely be the focus. Although I have had ideas about how to easily specify a simple GUI form that pops up when a Jeql script is run, to accept input from the user....

That doesn't mean that there might not be room for some utility code
that makes integration of JEQL into a GUI easier. For example, you
might have a wrapper that returns the results of a JEQL query
displayed in a simple JTable, or if the results or spatial, in a
LayerViewPanel.

 Maybe then we can
talk about integrating a DXF reader/writer for JEQL.

Ok...  Is there any possibility of just using the DXF code from OJ?

[Sunburned Surveyor]

Martin,

You wrote:

Almost... The syntax is:

<commmand-name> [ <default-parameter-expression> ] { <param-name>:
<expression> }

(where [ ] means "optional" and { } means "0 or more" )

(I like your term "action" tho - but too late to change now!)

Command makes sense too. I'm a little confused about
<default-parameter-expression>. What does that mean? Is it just a way
to specify a required parameter?

You wrote:

"Although I have had ideas about how to easily specify a simple GUI
form that pops up when a Jeql script is run, to accept input from the
user...."

That would be cool. Would you automatically create a form for any
number of parameters using something like GridLayout? Or would you
allow a Swing class to be passed to JEQL? Inkscape creates a simple
form for user extension scripts based on a text file describing the
input parameters. I seem to remember there being a class in JUMP that
did something similar, but I can't remember what is was called.

You wrote: "Ok... Is there any possibility of just using the DXF code from OJ?"

This depends on three (3) things:

[1] Your understanding of French. All the Javadoc comments for the
OpenJUMP DXF plug-in are in French, if I remember correctly. However,
if your knowledege of Java is sufficient, you can probably understand
the code without the Javadoc comments. I'm not a good enough
programmer for that.

[2] The level of access you want to the DXF file. If I remember
correctly the DDXF plug-in provides rather "high-level" access to DXF
files, which can be somewhat limiting. Perhaps it is possible to
extract and use the lower level functionality, but I didn't know
enough French to determine this. :]

[3] The type of entities you want to be able to access from the DXF
file. For example, the OpenJUMP plug-in only accesses points and
polylings if I remember correctly. I'd like to be able to get to other
DXF entities like MTEXT, DTEXT, Viewports, and the Layer Table. Most
of this isn't necessary if all you want is simple features from the
lines or polylines in the DXF file. For that the OJ plug-in is
probably sufficient. For much of my work it will not be good enough.

It also seems that I had trouble getting the plug-in to recognize
closed polylines as polygons, or to leave closed polylines as
lineStrings. I can't remember which one.

Landon

[Martin Davis]

On Wed, Mar 26, 2008 at 7:44 AM, Sunburned Surveyor <sunburned...@gmail.com> wrote:

Command makes sense too. I'm a little confused about
<default-parameter-expression>. What does that mean? Is it just a way
to specify a required parameter?

default parameters are just a way to supply a commonly used parameter without the overhead of giving a parameter name as well.  For instance, this lets you write Print x rather than Print object: x

That would be cool. Would you automatically create a form for any
number of parameters using something like GridLayout? Or would you
allow a Swing class to be passed to JEQL? Inkscape creates a simple
form for user extension scripts based on a text file describing the
input parameters. I seem to remember there being a class in JUMP that
did something similar, but I can't remember what is was called.

I think I'd let you spec out a simple form using a table with formatting info, and then pop up a Swing dialog with that formatting.  You could then get the input via special functions.  This is very like the MultiInputDialog in JUMP - which is very useful.

You wrote: "Ok...  Is there any possibility of just using the DXF code from OJ?"

This depends on three (3) things:

Ok, thanks for the info - not quite as easy as I'd hoped then.

M

[Sunburned Surveyor]

Martin,

You wrote: "default parameters are just a way to supply a commonly

used parameter without the overhead of giving a parameter name as
well. For instance, this lets you write Print x rather than Print
object: x"

This makes perfect sense now. I think that I am one the way to
understanding the basic JEQL command syntax. :]

You wrote: "This is very like the MultiInputDialog in JUMP - which is
very useful."

That is the class I was thinking of.

Martin wrote: "Ok, thanks for the info - not quite as easy as I'd hoped then."

You are Canadian, I thought maybe you spoke French. :]

Landon

[Martin Davis]

On Wed, Mar 26, 2008 at 9:32 AM, Sunburned Surveyor <sunburned...@gmail.com> wrote:

You are Canadian, I thought maybe you spoke French. :]

Touche!  My french is barely passable - but my wife is une Quebecoise, so she's fluent.  So I could probably figure it out....