openGL ffi, some basic SDL ffi, automatic FFI

[Jasper]

Wrote a basic C header parser in Common Lisp:
https://github.com/o-jasper/parse-c-header
and made a little C ffi making helper tool with it, and used it to make
the openGL ffi. (and probably in the future other ffi's)

Tbh kindah sucks that i feel i need to write a parser for this, and
that i cannot find a decent lib/program that gets the data with
minimal parsing and/or interpreting needed.. But i don't like doing
things manually so i made it.

For examples and (real-time) plotting
https://github.com/o-jasper/julia-glplot

I can translate the parser and ffi generator into Julia if desired.

[Jeffrey Sarnoff]

There is widespread opportunity (and at least one pressing need) for julia code that autogenerates a proper julia file used to interface with a C library by digesting the library's header file (which may include other present headers) and providing julia ccall wrappers and julia types to mirror any C structs or pointers to structs appearing as C typedefs.  Let me know if you decide to pursue it -- and need a test case.

[Tim Holy]

Hi Jasper,

This looks potentially quite interesting. I cloned the two repositories, but I
was not able to get things to run. Is it possible that your documentation
might omit some important steps in getting this set up?

Here's what I did:
Clone the repositories
Add the path as suggested in the readme for julia-glplot, and the path as
suggested in the readme for parse-c-headers/src/julia-src
julia> load("continuous_plot_gl.j")
Histogram not defined
in load at util.jl:230
in load at util.jl:242
at /tmp/julia-glplot/julia-glplot/continuous_plot_gl.j:119
in load at util.jl:253

From the bash prompt:
$ sh run.sh
real_path: No such file or directory
in load at util.jl:253
in process_options at client.jl:178
in _start at client.jl:227

It's pretty easy to guess that some of the dependencies aren't being loaded.
Might I suggest putting require("histogram.j") and any other dependencies in
any top-level functions (those meant to be run from the command line)?

Other things that would be good to clarify:
Does this run on Linux? OSX? I don't even know if it's supposed to work for my
platform.
Does this depend on external libraries? For example, in the parse-c-headers
you mention SDL. Is this a dependency? Many people won't have it installed, so
it would be worth mentioning such things.
How about something that says, "If you want to try it out, set these paths,
start Julia, and type <XYZ> at the prompt"? It would be lovely if I knew which
of these functions could be expected to produce some fun output.
Is there anything besides adding the path that needs to be done for the parse-
c repository? For example, are there functions there that I need to run before
I can expect the glplot functions to work?

There may be other points of potential confusion, but these are a start.

--Tim

[Jasper]

Ok, thanks for trying it, and moreover, reporting about it. I messed up;
1) when using it, i didn't notice i was depending paths relative to
`pwd`. 2) the `util` directory was missing. Both should be fixed in
688e452, there is an example `doc/juliarc_part.jl` file now.(basically
the minimal one i tried)

Note that run.sh is basically just for convenience. It requires one
argument -the file to run- which has a `run_this()` function. Files with
a `run_this()` also `load` all the stuff they need.(files without
`run_this()` can't be run, they're just pots of stuff.) In the new
version it shouldn't matter where you run `run.sh` from, it should just
matter which file you point too.. So please:

* pull julia-glplot again
* try `julia-glplot/run.sh julia-glplot/test/continuousplot.j`
* if not maybe check `~/.juliarc.jl`, not that it should not
say /julia-glplot/julia-glplot/ (just once) and try again

I'll get a setup for running with running/testing the minimum possible
configuration and outside the context i using in the future. I should
also look for any info on how to package stuff standardly..(I hope
we'll eventually have something asdf-like but better)

It should run on Linux. Not sure about the others. The weak spots
probably are whether `dlopen` can find what it needs, and whether the
SDL init stuff is compatible.

About converting the parsing Julia: there shouldn't be anything
preventing it. So probably i'll do it.

[Tim Holy]

$ julia-glplot/run.sh julia-glplot/test/continuousplot.j
could not load module
/home/jasper/proj/julia/julia_c/sdl_bad_utils/init_stuff.so:
/home/jasper/proj/julia/julia_c/sdl_bad_utils/init_stuff.so: cannot open shared
object file: No such file or directory
in dlopen at base.jl:116

in load at util.jl:230
in load at util.jl:230
in load at util.jl:242

in process_options at client.jl:178
in _start at client.jl:227

at /tmp/parse-c-header/src/julia-src/sdl_bad_utils/init_stuff.j:4

in load at util.jl:230
in load at util.jl:230
in load at util.jl:242

in process_options at client.jl:178
in _start at client.jl:227

at /tmp/julia-glplot/julia-glplot/test/continuousplot.j:7

in load at util.jl:253
in process_options at client.jl:178
in _start at client.jl:227

Note there is no user "jasper" on my laptop :-).

Can I recommend that you give it a try on some machine that you didn't use to
develop this, perhaps under a different username?

--Tim

[Jasper]

Now run parse-c-header/src/julia-src/sdl_bad_utils/compile.sh from its
directory. It will make those .so files.

It does say that so, but indeed not clear enough.

[Tim Holy]

Still fails. If you continue to be interested in trying to get this ready for
testing, first do a recursive grep for "jasper" in both source trees. It sure
looks like you're using hard-coded paths to your personal home directory.

I've exceeded my quota on this for now, you may need to find other testers.
Sorry.

--Tim

[Jasper]

I am such an idiot.. the `dlopen`s of `sdl_bad_utils` do refer to it.
Not sure how to get around that yet.. How do you get the directory the
file itself is loaded from? Or how do you tell julia to look in some
particular directory?(relative to the directory that is being evaluated)
Setting LD_LIBRARY_PATH works, and i am setting it, but that is a bad
fix...

Apologies for being so messy.

[Jeffrey Sarnoff]

To find out something about Julia when you are not sure where to look, there are three places to check first:
using the Search for topics field in julia-dev, using grep in julia/base, julia/extras, julia/examples
and searching the docs with google:
   to discover what may be said about directory use,  enter this into goole:  +(" dir" OR directory) site:http://docs.julialang.org/en/latest

At present there is no JULIA_HOME environment variable to check, and I do not know if there is a function or var that tells you Julia's install dir.
One approach is to write file pathnames relative to Julia's install directory and tell the user your pathnames expect that julia is run within the install dir.
Specify file directories relative to the Julia's base directory (the one called "julia" with subdirectories that include "julia/base" and "julia/extras" ).
So to load the file named "poly.jl" from the directory "julia/extras", in your julia code write: load("./extras/poly.jl");

Put your files in new subdirectory, e.g. "./julia/extras/jasper/".  Also look at ./base/file.jl, and from within julia: cwd().
You can ask the user to provide the filepathname to a targeted header file:

function getHeaderFilepath()
         print("\n Which header file (the full path/name)? ");
         flush(stdout_stream);
         pathname_of_header_file = readuntil(stdin_stream,'\n')[1:(end-1)]
         # check that it exists, if so, do something
end

repost here when you have it the way you want and a friend has checked that it loads works as expected on another machine

[Jameson Nash]

Adding paths to the shared library search process is currently only possible by setting LD_LIBRARY_PATH beforehand. (Unfortunately, it is different from loading Julia code.) I've wanted to move that code into Julia to make it more flexible, but I've been rather preoccupied with the libuv branch. You could have the user drop the shared object files in julia/usr/lib, or provide relative paths to them from the current working directory, or just:

dlopen("$JULIA_HOME/../../extras/jasper/lib/libsomething") for an absolute path

 (note that extras and lib are automatically on the search path so load("poly.jl") should typically work, except where poly.jl already exists in the current search directory).

See the Julia variables JULIA_HOME and LOAD_PATH to find Julia's install directory and include search paths, respectively.

-Jameson

--
 
 
 

[Jeffrey Sarnoff]

I cloned julia while in "/home/jas/Public" so the git copy went into "/home/jas/Public/julia" and that directory is where I need to be to do git stuff cleanly.
I had noticed that make put stuff into the ./julia/usr directory tree; and so  JULIA_HOME == "/home/jas/Public/julia/usr/bin" makes sense.
Is it safe and portable to identify the local root of a cloned Julia installation as "($JULIA_HOME)/../../" ?
When released, will user installs have a local julia root, or will it go into /bin /lib places? and what about Windozer?

[Jasper]

Had a look round and Jameson is correct afaik.(.so files were the
problem at the end, not `load`) Well, maybe nearly, i currently use:(and
have pushed on git, but given history, i _won't_ say it works for others
yet, i'll test it more later.)

dlopen(find_in_path("sdl_bad_utils/sdl_event.so"))

Which basically 'dlopen as it would load'. `load` uses it internally.
And load can then be given links with LOAD_PATH, so a project basically
needs only one entry in .juliarc.(Which imho is the maximum number of
entries needed that is still good.. Zero even better, by being able to
indicate directories as containing julia projects?)

It kindah makes sense to do it 'as load'. Maybe i could venture as far
as saying `loadso` might be a nice function to have. But it makes less
sense than, in the future, having a system that expresses dependencies.
(Refraining from saying too much here.)