[felix-lang/felix] 7729af: Makefile fixes:

[GitHub]

Branch: refs/heads/fix-makefile
Home: https://github.com/felix-lang/felix
Commit: 7729af62a08e5841c9904640c88355cfe5604133
https://github.com/felix-lang/felix/commit/7729af62a08e5841c9904640c88355cfe5604133
Author: Martin DeMello <martin...@gmail.com>
Date: 2014-08-27 (Wed, 27 Aug 2014)

Changed paths:
M Makefile

Log Message:
-----------
Makefile fixes:
* Use $PREFIX ($INSTALLDIR was being set but ignored)
* Don't mix up $HOME/.felix with the install process

[Martin DeMello]

A few fixes to the `make install` process:

1.  The makefile defined a $INSTALLDIR, but the actual install: target hardcoded /usr/local/lib/felix. I switched to the more conventional $PREFIX.

2. User should call "sudo make install" rather than "make install" calling "sudo". This is also pretty conventional

3. We should not be touching $HOME/.felix during the install process; it's an entirely separate concern

martin

--
You received this message because you are subscribed to the Google Groups "Felix Language" group.
To unsubscribe from this group and stop receiving emails from it, send an email to felix-languag...@googlegroups.com.
To post to this group, send email to felix-l...@googlegroups.com.
Visit this group at http://groups.google.com/group/felix-language.
For more options, visit https://groups.google.com/d/optout.

[Martin DeMello]

hm - just realised that won't work due to cache issues. it's really weird to have global installation and a user-owned cache tangled up that way; i think cleaning up the build and install system would be a very high reward-to-effort thing to do. i'm willing to put in some work to help with it.

martin

[john skaller]

On 28/08/2014, at 3:59 PM, Martin DeMello wrote:

> A few fixes to the `make install` process:

OK, but I'm having a hard time figuring out how to pull your changes ..

I would rather give you commit rights to the main repository than
stuff about with pull requests.

I have a problem with GitHub it tries to be smart and logs me
in as the wrong user all the time (two users with the same email
address .. one is for me and the other is for committing
from the server).

> 1. The makefile defined a $INSTALLDIR, but the actual install: target hardcoded /usr/local/lib/felix. I switched to the more conventional $PREFIX.

Felix itself uses INSTALL_DIR (with an underscore) internally.
The variable in the makefile is independent of this of course.

however usually on Unix systems PREFIX will be

/usr
/usr/local

and the lib, bin, and include directories used will be subdirectories of PREFIX.

Felix doesn't use this: everything goes in "lib" like with Python EXCEPT
for the "flx" binary (which has to go on the PATH so it will run).

>
> 2. User should call "sudo make install" rather than "make install" calling "sudo". This is also pretty conventional
>
> 3. We should not be touching $HOME/.felix during the install process; it's an entirely separate concern

Well you can see the reason. Felix builds a test program to trigger building
standard library in the cache, as part of the install process.

It is absolutely essential this not be done as root.

however I do agree with both your points.

No damage is done removing the writes to the cache.
It will happen on the first use of "flx" anyhow.


Note also there is a new mechanism for using Felix, which is
"work in progress". This involves

flx --felix=config.fpc ....

With this machinery all the directories searched or used can be individually
specified.

The reason is partly because I want to add "user package" to the system
and partly because it is very inconvenient to have to "install" Felix in a place
only root has access to.

With this machinery, a traditional "Debian" like install should be possible:
binaries in /usr/bin, libs in /usr/lib, include files in /usr/include/felix or whatever.

However it's not ready for the big time yet :)

--
john skaller
ska...@users.sourceforge.net
http://felix-lang.org

[john skaller]

On 28/08/2014, at 6:54 PM, john skaller wrote:

>
> On 28/08/2014, at 3:59 PM, Martin DeMello wrote:
>
>> A few fixes to the `make install` process:
>
> OK, but I'm having a hard time figuring out how to pull your changes ..

Let me try to fix this according to your suggestion.

[Martin DeMello]

I pushed it to a branch in the main repository :) the github pull request was just so that there would be a convenient place to add comments to the change.

I think the ideal thing to do would actually be to have a way to install felix purely as user, to a directory within $HOME (either one of the user's choosing, or to ~/.felix/bin) that can be added to $PATH. And as an entirely separate issue, add a felix --init-cache or something like that, rather than having it be part of the install process.

martin

[john skaller]

On 28/08/2014, at 7:16 PM, Martin DeMello wrote:

> I pushed it to a branch in the main repository :) the github pull request was just so that there would be a convenient place to add comments to the change.
>
> I think the ideal thing to do would actually be to have a way to install felix purely as user, to a directory within $HOME (either one of the user's choosing, or to ~/.felix/bin) that can be added to $PATH. And as an entirely separate issue, add a felix --init-cache or something like that, rather than having it be part of the install process.

Done .. added "make-cache".

As noted, however, this is not enough.

Examine

build/release/host/lib/plat/config.flx

to see why: the install directory is referred to there.

Now .. I am trying to entirely eliminate this file
precisely because it makes the system hard to move.
Certainly the directory there can be overridden in "flx"
using an environment variable, or --test=dir option.
Or the new --felix=layout.fpc option (when that works).

In the meantime the issue is this: when you say

flx hello.flx

you expect it to work, and lacking any other data, the
place to find the libraries HAS to be hardcoded somewhere.

You may wonder, where does this file come from?

The answer is found in

lpsrc/flx_maker.pak

which uses Python to generate the config file.
That in turn gets data from fbuild which allows

fbuild/fbuild-light --prefix=location ....

but the makefile doesn't use that.

Since one does "make build" and "make install" separately its
hard to see how to fix this at the moment.

the usual way is to set the install directory in

./configure --prefix=location

and then the Makefile obeys that.

[john skaller]

On 28/08/2014, at 7:40 PM, john skaller wrote:

>
> As noted, however, this is not enough.
>
> Examine
>
> build/release/host/lib/plat/config.flx
>
> to see why: the install directory is referred to there.
>
> Now .. I am trying to entirely eliminate this file
> precisely because it makes the system hard to move.

So here's the plan: please COMMENT if you have any thoughts.

I will change the config.flx file to look for data in

$HOME/.felix/felix.fpc

If found this would override the hard coded location.
So plain

flx hello.flx

will work, using that file to locate the installation.

Ultimately, the FLX_INSTALL_DIR should cease to exist:
a layout file should always be used.

[USERPROFILE is used on Windows I think, instead of HOME]

This will make the system adaptable, although some arcane knowledge
is required to get it to work.

It would be good to eventually *enforce* the layout manager,
by removing the generated plat/config.flx entirely (and fixing
any breakages).

[Martin DeMello]

That sounds like an excellent change! Personally as a linux user I'm happy to add things to my $PATH to find a binary, but would prefer the binary had some fixed, internal way to find the rest of its stuff, either by writing to a config file in a known location or at the very worst by a well-documented set of environment variables I would need to set in my .bashrc.

martin

[john skaller]

On 30/08/2014, at 5:53 AM, Martin DeMello wrote:

> That sounds like an excellent change! Personally as a linux user I'm happy to add things to my $PATH to find a binary, but would prefer the binary had some fixed, internal way to find the rest of its stuff, either by writing to a config file in a known location or at the very worst by a well-documented set of environment variables I would need to set in my .bashrc.

The latest commit does the following:

The host/plat/config.flx file is gone.

This was a platform dependent library file which encoded the
configured installation directory.

Instead there is a platform *Independent* file

share/lib/plat/config.flx

containing a function which does the following:

1. FLX_INSTALL_DIR is set to /usr/local/lib/felix/felix-<version>
[This is a bug, it should be felix-latest IMHO]

2. if $HOME exists and $HOME/.felix/config/felix.fpc exists
and it contains the a line starting with exactly

'FLX_INSTALL_DIR: '

excluding the quote makes, the rest of the line is used to set
the variable FLX_INSTALL_DIR.

If the above mentioned file doesn't exist a warning is printed.
I think the file will become required and the warning will change
to a program abort but we'll see.


3. If the environment variable FLX_INSTALL_DIR is non-empty
string its value is used to set the variable FLX_INSTALL_DIR

The function returns a record:

(
FLX_INSTALL_DIR=FLX_INSTALL_DIR,
FLX_TARGET_SUBDIR="host"
)

The client program, for example flx, can then override the
returned value in some way it likes. At present flx has two options:

4. flx --test=dirname ...

resets the install dir to dirname. I use this testing, often saying

flx --test=build/release ...

5. flx --felix=filename

This reads a configuration file which may override the install directory.
Typically this would be

$HOME/.felix/config/felix.fpc

i.e. the same file used by the config function.

I could think about looking for ./felix.fpc as well,
or perhaps ./project.fpc so each directory you're
working in could have a "project" file.

================================================

So: THIS IS A TEMPORARY HACK. It is part of an "Agile"
transition to use a different layout mechanism.

With these changes you can use

make PREFIX=/home/myself install

to do an installation, for example. Being able to install felix somewhere
that does not require administrator privileges is important.

In the new system there is no "installation directory".
instead the various "parts" of Felix can be individually specified.

There will be a "standard layout" which simplifies this,
roughly the existing layout using FLX_INSTALL_DIR.

There are several reasons for this.

First, if you have a read only shared Felix system, perhaps on
an enterprise wide server, you still may want to build a target
which isn't supported. For example, I myself want to build
a clang and gcc based target so I can check both work.
Someone else may want to build a cross compilation target.

At present the target is a subdirectory of the install directory.
So you cannot build a new target privately without copying
the install directory (yea, on Unix you could use a link).

Another problem is the standard library is cached, and if you change
anything the whole library has to be recompiled. This takes
ages, and I myself want to switch between the build/release
version for testing and the installed version. So I want to allow
for independently specifiable cache locations to avoid rebuilding
the cache every time I change something.

However there is another, much more important reason:

PACKAGES

Where do we put them? The answer in the default layout is probably

/usr/local/felix/site-packages

following Python. Or

/usr/local/felix/share

or something .. :)

But of course .. YOU cannot write there, only the admin
can. So we need a way to specify a search path for libraries.
[I mean, not every time on the command line!!]

Also whilst Felix code is read only text files, C++ code may want
objects, object libraries, and shared libraries to be built and these,
like the core RTL, need to be split off away from the universally
shared source code, so that each target gets its own binaries.

Its important packages do not live in the core Felix directory,
because when you upgrade Felix (without a version change)
the directory is deleted before installation (to make sure it is clean).
OTOH when you install a new version, the directory won't have the
packages in it and you don't want to have to install them all again.

So the packages should live outside any particular Felix version.
Unless they're core packages.

To make all this work I want a single point layout control file.
Finding and specifying it will be a PITA and a source of error
and confusion but once it is found everything can be made explicit.


[IF felix every gets installed into /lib then the config would live in /etc
on Unix. in fact I have my eye on the idea that Felix script should
be used for Unix init processing. The Sys V method with bash is
unmaintainable, upstart is a bitch, etc. Having a type safe language
to control system init makes sense. Having a fully compiled to binary
language for a fast secure start does too]

[Martin DeMello]

On Fri, Aug 29, 2014 at 4:28 PM, john skaller <ska...@users.sourceforge.net> wrote:

To make all this work I want a single point layout control file.
Finding and specifying it will be a PITA and a source of error
and confusion but once it is found everything can be made explicit.


[IF felix every gets installed into /lib then the config would live in /etc
on Unix. in fact I have my eye on the idea that Felix script should
be used for Unix init processing. The Sys V method with bash is
unmaintainable, upstart is a bitch, etc. Having a type safe language
to control system init makes sense. Having a fully compiled to binary
language for a fast secure start does too]

Another idea is for the config file to *always* live in $HOME/.felix/<somewhere>, if necessary copied in from a fixed (written once at install time and never modified thereafter) file in /etc/felix, but maintained in .felix thereafter.

Actually, my somewhat more drastic inclination (you will probably not agree!) is to get rid of the ability to use felix as a systemwide scripting language altogether, embrace the fact that it generates native executables, and let the default be for developers to always install felix as user, set up their dev environment, and use it to create executables which need no reference to the felix compiler to use them. Anecdotally, I have never had a good experience developing in ocaml until I gave up on the system install and switched to exclusively using opam as user, with both the compiler itself and all the installed packages maintained under $HOME/.opam. As a developer this model works beautifully, and realistically, the target audience for felix is 100% developers now and for the foreseeable future.

martin

[john skaller]

On 30/08/2014, at 9:53 AM, Martin DeMello wrote:

> Another idea is for the config file to *always* live in $HOME/.felix/<somewhere>, if necessary copied in from a fixed (written once at install time and never modified thereafter) file in /etc/felix, but maintained in .felix thereafter.

Well its not "another" idea: I did write this:

"""
2. if $HOME exists and $HOME/.felix/config/felix.fpc exists
and it contains the a line starting with exactly

'FLX_INSTALL_DIR: '

excluding the quote makes, the rest of the line is used to set
the variable FLX_INSTALL_DIR.

If the above mentioned file doesn't exist a warning is printed.
I think the file will become required and the warning will change
to a program abort but we'll see.
"""

>

> Actually, my somewhat more drastic inclination (you will probably not agree!) is to get rid of the ability to use felix as a systemwide scripting language altogether, embrace the fact that it generates native executables, and let the default be for developers to always install felix as user, set up their dev environment, and use it to create executables which need no reference to the felix compiler to use them.

You can do this now. If you use --static switch, you get an executable. At present it will
be hidden in the cache, but you can change that with the -od . switch:

flx --static -c-od . hello.flx

will produce an executable hello (or hello.exe on Windows) in the current directory.

However, remember Felix is actually designed as a *library* generator not a
program generator. The fact it can generate programs is actually a minor side
effect of the design.A program is the side effect of library initialisation code,
when the library is combined with "flx_run" mainline to run it.

As an example, the webserver flx_web is built as a short program
which loads a system of plugin libraries. So we need to be able to
build these plugin libraries: they're shared libraries not programs.

Actually dflx_web loads these with dlopen(). But flx_web uses
an emulator to trick the system into using statically linked modules
instead, so flx_web is a self contained executable.

The flx program does this too: there's a dflx which loads toolchains
dynamically, but a flx which finds common ones like gcc and clang
statically linked (so operation is reliable).

FYI in both cases the program logic is unchanged. The program
thinks it is doing dynamic loading. However a wrapper is used
to "preload" certain libraries, which are static linked. So for any
program using plugins, you can freeze any set of plugins
into the executable with a suitable wrapper.

however there IS another problem with $HOME/.felix/config/felix.fpc just being
the single mandatory layout control file: Felix is designed to support cross compilation.

You will always need a "host" target for your local machine to build the
development tools, but you may also want to build other targets:

Android
iPhone

are obvious examples. Or if you're on Linux you may also want to build

Windows

using a gcc based cross compiler.

These are extreme examples, less extreme ones are building with both gcc
and clang, or building a debug and non-debug version of something,
or building a --usage=production version as well as a --usage=debug
version (use different optimisation switches).

Or what I do: I have an installed Felix AND one in build/release
that I want to test. The bootstrap process actually requires an
installed version to build the a new one, for obvious reasons.

Obviously with multiple targets a single control file is not enough.

but the bottom line is I do believe a single initial control file
is needed instead of a lot of hacks finding things.

Another example I just ran into: for the SDL based GUI to work
we need some fonts. Where these fonts live varies from
platform to platform. We can use the configuration database
at run time to find them .. but where is the configuration
database? if you hate a font, you want to be able to run a
GUI program with a font you like .. :)

> Anecdotally, I have never had a good experience developing in ocaml until I gave up on the system install and switched to exclusively using opam as user, with both the compiler itself and all the installed packages maintained under $HOME/.opam. As a developer this model works beautifully, and realistically, the target audience for felix is 100% developers now and for the foreseeable future.

Of course, Felix is a programming language and development system, so the target
is clearly developers.

And I tend to agree that to make the whole thing work nicely one needs a
package manager.

Mike Maui's original scoop tool was a good start, but it couldn't really handle
the separation between the platform independent sources and target
relative binaries, nor was there really any systematic way to do things
like compile C/C++ parts of packages.

There still isn't but I'm working on it :)

[john skaller]

On 30/08/2014, at 9:53 AM, Martin DeMello wrote:

> Actually, my somewhat more drastic inclination (you will probably not agree!) is to get rid of the ability to use felix as a systemwide scripting language altogether, embrace the fact that it generates native executables, and let the default be for developers to always install felix as user, set up their dev environment, and use it to create executables which need no reference to the felix compiler to use them.

Originally the idea is that you can write *platform independent* programs and just
run them. No switches. No build steps. No conditional compilation for Windows
vs Linux.

Actually Ocaml does the last thing quite well up to the limits of the
standard distro: the Felix compiler "just works" on both Windows and Unix
with the only work needed to support this in the compiler being some code
to handle filenames (slash separator vs slosh)

Anyhow if you look at the Felix build process there is quite a lot of using
flx to generate binary stuff and needing switches to control the build steps,
however the design tries to make the shell commands needed platform
independent. Thats why you do:

flx --static -c -od . hello.flx

The -od dirname flag is the same on Windows as Unix.

But then consider this, in "make test":

${BUILDROOT}/host/bin/flx --test=${BUILDROOT} \
--usage=prototype --expect --nonstop \
--indir=test/regress/rt --regex='.*\.flx' test

It's kind of useful you don't have to specify the output of the exe.
In fact in this case .. there's no --static .. so there's no exe, it actually
makes

testcase.dylib

on OSX, .dll on Windows and .so on linux. I really don't want to say
where the binaries go: they go in the cache, i just want to run
and forget.

So really, rather than "forget" the concept of running script,
it provides a useful starting point to use the language without
hassle.

Ultimately, yes, you will need to build components in steps.
Hopefully .. without auto* tools :)

[Martin DeMello]

On Fri, Aug 29, 2014 at 5:37 PM, john skaller <ska...@users.sourceforge.net> wrote:

However, remember Felix is actually designed as a *library* generator not a
program generator. The fact it can generate programs is actually a minor side
effect of the design.A program is the side effect of library initialisation code,
when the library is combined with "flx_run" mainline to run it.

But compiled felix libraries are indistinguishable from c++ libraries, right? In which case, why does felix need a systemwide language package manager? Let $HOME/.felix be the *only* supported place to install libraries into. What I'm saying is ruthlessly strip away unneeded features and concentrate on providing a really good experience for a single developer doing everything as user, in a single $HOME directory or mac/windows equivalent. This is, after all, felix's primary audience.

Anecdotally, I had some free time over the last couple of days and I thought I'd contribute by adding a much-needed examples/ directory to the felix repo, but in two evenings of trying, I've still not been able to get past

$ flx

Cache may be out of date due to grammar upgrade!

Grammar time stamp          =1409204024.000

Automaton.syntax time stamp =BIG BANG

Delete cache /home/mdemello/.felix/cache/text

Delete cache /home/mdemello/.felix/cache/binary

 

and flx --test=build/release does get past that, but rebuilds the stdlib every time. From the diagnostics, it successfully writes out

/home/mdemello/.felix/cache/binary/home/mdemello/code/felix/build/release/share/lib/grammar/grammar.files/syntax.automaton

every run, but then cannot read it back the next run:

Can't load automaton '/home/mdemello/.felix/cache/binary/home/mdemello/code/felix/build/release/share/lib/grammar/grammar.files/syntax.automaton'from disk: building!

This is simply not a good user experience, nor an encouragement to contribute if every time I try to write some code I end up wrestling with the ecosystem instead.

however there IS another problem with $HOME/.felix/config/felix.fpc just being
the single mandatory layout control file: Felix is designed to support cross compilation.

How about a per-project add-on config file (search from `pwd` upwards till you find one) for things like this? While supported, cross-compilation surely isn't the common-path use case, nor is fiddling with multiple versions of the compiler.

Another example I just ran into: for the SDL based GUI to work
we need some fonts. Where these fonts live varies from
platform to platform. We can use the configuration database
at run time to find them .. but where is the configuration
database? if you hate a font, you want to be able to run a
GUI program with a font you like .. :)

From what I understand, felix works via c++ code generation, right? So surely the time all these issues will need to be resolved is when you hand control over to the c++ compiler, rather than while felix is running. Document the api and let the developer configure font paths himself, don't try to do everything automatically. It's more important that I should be able to set this stuff up once for my project and then forget about it, than that felix tries to guess it for me - the difference is that felix needs to cope with every possibility of OS and environment, but development is typically done on a single machine with a fixed environment that developers can configure for themselves.

And I tend to agree that to make the whole thing work nicely one needs a
package manager.

Mike Maui's original scoop tool was a good start, but it couldn't really handle
the separation between the platform independent sources and target
relative binaries, nor was there really any systematic way to do things
like compile C/C++ parts of packages.

There still isn't but I'm working on it :)

 I think felix is trying to do too many things with too limited a supply of manpower. I'd suggest at least looking seriously at picking on an existing cross-platform package management system, and modifying felix's library/include search mechanism to interoperate with it.

martin

[Ryan]

I still wish it would be better advertised that you CAN generate static executables. Being able to run something and have it "just work," while also being able to generate highly optimized executables for distributing, is a major advantage.

BTW, I thought you could do:

$ flx -c --static -o myapp myapp.flx
$ ./myapp

>
>Ultimately, yes, you will need to build components in steps.
>Hopefully .. without auto* tools :)

Don't mention that name, or else I'll have endless nightmares about m4. CMake isn't really too much better (but, in comparison, it's heavenly).

>
>--
>john skaller
>ska...@users.sourceforge.net
>http://felix-lang.org

--
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

[john skaller]

On 30/08/2014, at 11:24 AM, Martin DeMello wrote:

> But compiled felix libraries are indistinguishable from c++ libraries, right?

Yes, with extern "C" entry points.

> In which case, why does felix need a systemwide language package manager?

For the Felix library code. For example suppose I want to install
a binding for GMP, or SDL. Where do I put it?

Underneath these are binary libraries and C header files,
but the binding code itself is Felix.

And at the moment quite a lot of code is pure Felix.

> Let $HOME/.felix be the *only* supported place to install libraries into.

Srean can't do that. His "home" account on university computer is
tiny. (though he could use a link).

> What I'm saying is ruthlessly strip away unneeded features and concentrate on providing a really good experience for a single developer doing everything as user, in a single $HOME directory or mac/windows equivalent. This is, after all, felix's primary audience.

Yes, but not one with any money :-)

So I need to consider enterprise wide deployment strategy too.

>
> Anecdotally, I had some free time over the last couple of days and I thought I'd contribute by adding a much-needed examples/ directory to the felix repo, but in two evenings of trying, I've still not been able to get past
>
> $ flx
> Cache may be out of date due to grammar upgrade!
> Grammar time stamp =1409204024.000
> Automaton.syntax time stamp =BIG BANG
> Delete cache /home/mdemello/.felix/cache/text
> Delete cache /home/mdemello/.felix/cache/binary
>
> and flx --test=build/release does get past that, but rebuilds the stdlib every time.

This will happen if you ran

sudo make install

on the old build system. root ends up owning the cache.

> From the diagnostics, it successfully writes out
>
> /home/mdemello/.felix/cache/binary/home/mdemello/code/felix/build/release/share/lib/grammar/grammar.files/syntax.automaton
>
> every run, but then cannot read it back the next run:
>
> Can't load automaton '/home/mdemello/.felix/cache/binary/home/mdemello/code/felix/build/release/share/lib/grammar/grammar.files/syntax.automaton'from disk: building!
>
> This is simply not a good user experience, nor an encouragement to contribute if every time I try to write some code I end up wrestling with the ecosystem instead.

Of course not, but it shouldn't happen. so I guess your user account doesn't have
permission to write the file: can you check if it is actually there?

> I think felix is trying to do too many things with too limited a supply of manpower.

Indeed. More man power is needed :-)

> I'd suggest at least looking seriously at picking on an existing cross-platform package management system, and modifying felix's library/include search mechanism to interoperate with it.

I'm not aware of any :-)

but the issue at the moment is simpler: to actually ALLOW the user to configure their
search paths. At the moment they can't always do that. That's what I'm trying to
enable at the moment.

[john skaller]

BTW: the original co-developer of Felix, RF, happens to be an iOS specialist.
And I'm interest in games.

In the game world .. simultaneous multi-platform development is
not only useful .. its worth a lot of money :)

Felix as a scripting language is there to write the development
scripts, Felix as a compiled language the actual games.

[john skaller]

On 30/08/2014, at 12:12 PM, Ryan wrote:

> BTW, I thought you could do:
>
> $ flx -c --static -o myapp myapp.flx
> $ ./myapp

You can. However

flx -c -od . myapp.flx

is better because it works on Windows too, there you get

myapp.exe

instead of Unix

myapp

You can also use

flx -c -ox myapp myapp.flx

which adds the .exe extension on windows automatically.

Similarly, -ox can be used whether the target is an object file:

.o on Unix
.obj on Windows

or a shared lib:

.so on Unix
.dylib on OSX
.dll on Windows


The -od option is most useful when batch compiling based on a regexp
and so you don't *know* the name of the file. So neither -o nor -ox will
work. But -od does work, because it only names te directory in which
to place the executable.

Look at test target in the make file. All the regression tests are run as
a single flx command.

All this is documented in detail. See

http://felix-lang.org/share/src/web/ref/tools.fdoc

http://felix-lang.org/share/src/web/ref/tools_flx_cli.fdoc

http://felix-lang.org/share/src/web/ref/tools_flx_separate_compilation_index.fdoc

[srean]

> Let $HOME/.felix be the *only* supported place to install libraries into.

Srean can't do that. His "home" account on university computer is
tiny. (though he could use a link).

They kicked me out of the university, so Its not a problem anymore, but others might be similarly constrained. So if it is not too much of a problem a link is a better idea.

[Martin DeMello]

On Fri, Aug 29, 2014 at 8:55 PM, john skaller <ska...@users.sourceforge.net> wrote:

>

> From the diagnostics, it successfully writes out
>
> /home/mdemello/.felix/cache/binary/home/mdemello/code/felix/build/release/share/lib/grammar/grammar.files/syntax.automaton
>
> every run, but then cannot read it back the next run:
>
> Can't load automaton '/home/mdemello/.felix/cache/binary/home/mdemello/code/felix/build/release/share/lib/grammar/grammar.files/syntax.automaton'from disk: building!
>
> This is simply not a good user experience, nor an encouragement to contribute if every time I try to write some code I end up wrestling with the ecosystem instead.

Of course not, but it shouldn't happen. so I guess your user account doesn't have
permission to write the file: can you check if it is actually there?

Yep, file was there. Not sure what the problem was earlier; at least running

$ cd ~/code/felix 

$ flx --test=build/release ...

works now. though doing it from a different directory and supplying the full path as --test=/home/mdemello/code/felix/build/release does not.

martin

[john skaller]

On 30/08/2014, at 5:34 PM, Martin DeMello wrote:
>
> Yep, file was there. Not sure what the problem was earlier; at least running
>
> $ cd ~/code/felix
> $ flx --test=build/release ...
>
> works now. though doing it from a different directory and supplying the full path as --test=/home/mdemello/code/felix/build/release does not.

It should, can you try that again with the full path and --debug-flx?

I just tried an absolute path and it worked (but my version is probably a bit newer).

~/felix>cat hello.flx
println$ "Hello";
assert false;
~/felix>build/release/host/bin/flx --test=/Users/johnskaller/felix/build/release hello
Setting FLX_INSTALL_DIR from /Users/johnskaller/.felix/config/felix.fpc
FLX_INSTALL_DIR='/usr/local/lib/felix/felix-latest'
Hello
terminate called after throwing an instance of 'flx::rtl::flx_assert_failure_t'
Shell terminated by signal SIGABRT
~/felix>cd ..
~>felix/build/release/host/bin/flx --test=/Users/johnskaller/felix/build/release felix/hello
Setting FLX_INSTALL_DIR from /Users/johnskaller/.felix/config/felix.fpc
FLX_INSTALL_DIR='/usr/local/lib/felix/felix-latest'
Hello
terminate called after throwing an instance of 'flx::rtl::flx_assert_failure_t'
Shell terminated by signal SIGABRT

<aside>
here, the FLX_INSTALL_DIR diagnostics show I have a $HOME/.felix/config/felix.fpc
file, however the --test overrides that (its a temporary diagnostic :)

The terminate message is a bug.
I don't know exactly what. It should show a Felix and C++ location
of the assertion failure. The right exception is thrown, doesn't look like
it is caught by the driver.
</aside>

I agree completely when the tools fail its a serious PITA for the programmer
and certainly doesn't encourage new players.

However .. there's only one way to debug the tools on many platforms in many
environments .. and thats for many people to try it out. One thing I simply cannot
do alone.

[Martin DeMello]

Blew my .felix away, trying again with the new PREFIX-enabled makefile. I got this far now:

mdemello@mdemello-glaptop2 (master) ~/code/felix

$ make PREFIX=/home/mdemello/opt install

rm -rf /home/mdemello/opt/lib/felix/felix-1.1.12

build/release/host/bin/flx_cp build/release/host '(.*)' /home/mdemello/opt/lib/felix/felix-1.1.12'/host/${1}'

build/release/host/bin/flx_cp build/release/share '(.*)' /home/mdemello/opt/lib/felix/felix-1.1.12'/share/${1}'

build/release/host/bin/flx_cp build/release '(VERSION)' /home/mdemello/opt/lib/felix/felix-1.1.12'/${1}'

build/release/host/bin/flx_cp build/release/host/bin '(flx)' /home/mdemello/opt/bin'/${1}'

build/release/host/bin/flx_cp src/ '(.*\.(c|cxx|cpp|h|hpp|flx|flxh|fdoc|fpc|ml|mli))' /home/mdemello/opt/lib/felix/felix-1.1.12'/share/src/${1}'

rm -f /home/mdemello/opt/lib/felix/felix-latest

ln -s felix-1.1.12 /home/mdemello/opt/lib/felix/felix-latest

mdemello@mdemello-glaptop2 (master) ~/code/felix

$ which flx

/home/mdemello/opt/bin/flx

mdemello@mdemello-glaptop2 (master) ~/code/felix

$ flx

Grammar include file '/usr/local/lib/felix/felix-1.1.12/share/lib/grammar/grammar.files' doesn't exist, exiting

Seeing if I can figure it out.

martin

[Martin DeMello]

Aha, figured it.

mdemello@mdemello-glaptop2 ~/code/test

$ flx hello.flx 

Grammar include file '/usr/local/lib/felix/felix-1.1.12/share/lib/grammar/grammar.files' doesn't exist, exiting

mdemello@mdemello-glaptop2 ~/code/test

$ echo $FELIX_INSTALL_DIR 

/home/mdemello/opt/lib/felix/felix-latest

mdemello@mdemello-glaptop2 ~/code/test

$ flx --test=${FELIX_INSTALL_DIR} hello.flx

hello world

[john skaller]

On 30/08/2014, at 7:43 PM, Martin DeMello wrote:

> Aha, figured it.
>
> mdemello@mdemello-glaptop2 ~/code/test
> $ flx hello.flx
> Grammar include file '/usr/local/lib/felix/felix-1.1.12/share/lib/grammar/grammar.files' doesn't exist, exiting
>
> mdemello@mdemello-glaptop2 ~/code/test
> $ echo $FELIX_INSTALL_DIR
> /home/mdemello/opt/lib/felix/felix-latest
>
> mdemello@mdemello-glaptop2 ~/code/test
> $ flx --test=${FELIX_INSTALL_DIR} hello.flx
> hello world

Use FLX_INSTALL_DIR

[Martin DeMello]

On Sat, Aug 30, 2014 at 2:48 AM, john skaller <ska...@users.sourceforge.net> wrote:

On 30/08/2014, at 7:43 PM, Martin DeMello wrote:

> Aha, figured it.
>
> mdemello@mdemello-glaptop2 ~/code/test
> $ flx hello.flx
> Grammar include file '/usr/local/lib/felix/felix-1.1.12/share/lib/grammar/grammar.files' doesn't exist, exiting
>
> mdemello@mdemello-glaptop2 ~/code/test
> $ echo $FELIX_INSTALL_DIR
> /home/mdemello/opt/lib/felix/felix-latest
>
> mdemello@mdemello-glaptop2 ~/code/test
> $ flx --test=${FELIX_INSTALL_DIR} hello.flx
> hello world

Use FLX_INSTALL_DIR

doh :) was staring me in the face all along!

martin

[john skaller]

On 30/08/2014, at 7:58 PM, Martin DeMello wrote:

> Use FLX_INSTALL_DIR
>
> doh :) was staring me in the face all along!

There's another way, which is partly working:

flx --felix=/home/me/.felix/config/felix.fpc ...

This is currently a bit of a hack. For details of flx command options see:

src/lib/std/felix/flx/flx_cmdopt.flx

There are a list of parameters you can set, see "setup-from-file".
The install dir can be set with

install-dir: dirname

heh! Not FLX_INSTALL_DIR :-)

Basically all that stuff is going to move into the config function.

[john skaller]

So the next patch, the config will return two independent directories:

FLX_INSTALL_DIR
FLX_TARGET_DIR

The first is mainly to get

FLX_INSTALL_DIR/share

and the next change will be to eliminate the FLX_INSTALL_DIR variable.
You can still set the FLX_INSTALL_DIR, but it will set

FLX_SHARE_DIR = FLX_INSTALL_DIR/share
FLX_TARGET_DIR=FLX_INSTALL_DIR/FLX_TARGET_SUBDIR

so it is basically just a macro. In turn, these new symbols will be
eliminated, that is, setting them just sets subsidiary variables
that are *actually* required to run flx, such as search paths for the RTL
and Felix, the grammar location, etc etc.

However the next change after that will be to allow *multiple* share dirs.
That's so we can then do this:

felix/felix-latest/share/lib/ // standard library
felix/packages/share/addon // an addon library

or something similar. This way you can just set the paths, the standard
layout will just be a convention, not something enforced.

Of course there are more variables to consider, eg cache location,
C++ search paths, static linker search paths, load time linkage
and run time linkage, location of fonts, etc etc..

So in the transition process if you use a non-standard layout some
thing may break. Or worse .. they may work, but not as you expected.
[I.e. silently use the wrong version of something]

bear with me: a full recompile on my Mac takes about 6 hours.

[john skaller]

BTW: its never been necessary to install Felix. You can just work directly
off build/release, or, copy build release anywhere you like. All you have
to do is say:

flx --test=dirname

to use this, or

export FLX_INSTALL_DIR=dirname

(and of course make sure flx is on your PATH, either copy
build/release/host/bin/flx to where you want it or add
build/release/host/bin to your PATH).

Some Felix binaries require dynamic loading. For these you must
also set LD_LIBRARY_PATH to build/release/host/lib/rtl.
If you use flx to run the binary it will do that for you.
You can do

flx --test=build/release --run-only executable

for this to work.

most of the flx options can be found by

flx --help

You can find the location which flx will use for the install by

flx --where

[Ignore the FLX_INSTALL_DIR etc showing in the current repository code
as these are just diagnostics]

just FYI: when i designed the install method I made a deliberate
decision that /usr/local/lib/felix was where it would install on
all platforms (even Windows). just so it would be easier to help people
debug things. Put stuff in a non-standard space and half the email
conversation is trying to find out where things are rather than sort
out the problem.

[john skaller]

OK, so I think the new setup if finally working.
config.flx is moved to

lib/std/felix/config.flx

and it supplies two variables:

FLX_SHARE_DIR
FLX_TARGET_DIR

to its client. Now, there is one more critical directory: the cache.
At present that's calculated by

lib/std/felix/flx/flx_profile.flx

from $HOME etc. In fact it calculates HOME. Actually the core directory
should be

FLX_HOME_DIR

which would be

$HOME/.felix

by default. So config should use this (instead of doing its own calculation).
The question is: why is config in std/felix,whilst HOME is in std/felix/flx?

Who needs to know FLX_SHARE/TARGET_DIR? Who needs FLX_HOME_DIR?

Obviously "flx" needs these things. However applications may need some stuff
too. For a start, LD_LIBRARY_PATH needs to be set to get plugins and shared
libs. If plugins live elsewhere, we need that. The webserver tracks teh installation
too and so do the build tools.

The GUI needs a standard font. From where? From FLX_HOME_DIR/config?
Or from the TARGET database?

[john skaller]

Ok so I have progressed the layout control further but now I
have a conceptual problem I need help with.

In principle, the layout is a set of directories specified
as absolute path names (and perhaps some data files as well).

That's the fine grained specification of a layout.

However no one wants to specify all those files.

So there are some high level control variables like

FLX_INSTALL_DIR

you can use instead. If you set that variable it sets

FLX_SHARE_DIR = FLX_INSTALL_DIR/ "share"

and then something like:

FLX_LIB_DIR = FLX_SHARE_DIR/"lib"

etc. In other words, there is a cascade of settings. You can then
override one of the lower levels variables, but the ORDER matters:
if you set FLX_LIB_DIR then FLX_INSTALL_DIR the latter would cascade
down and clobber the former.

Now the problem starts. Initially, FLX_INSTALL_DIR is defaulted
to /usr/local/lib/felix/felix-latest and the cascade is done, however
there are TWO ways at least to change this:

1. with an environment variable FLX_INSTALL_DIR
2. with an entry in a control file (HOME/.felix/config/felix.fpc)

At present, the environment variable is applied in the first cascade:

FLX_INSTALL_DIR = getenv ("FLX_INSTALL_DIR", "/usr/local ...);

and then the control file is read.

I hope that's OK. However now things get tricky!

By default the target subdirectory of FLX_INSTALL_DIR is "host".
But suppose you want to use a different target.
So in your control file you can say:

FLX_TARGET_DIR: /home/skaller/mytarget

OK, so now the problem is, when you use a different target,
you probably want to use a different cache. So when you compile
for two different targets the cache doesn't get rebuilt all the time.

So, we can look for

FLX_TARGET_DIR/"config"/"felix.fpc"

and process that. So now, we've switched the config dir from

HOME/.felix/config

to

TARGET/config

in effect. Things are looking a bit recursive now .. :)

It gets worse. This is just the global config processing.
"flx" will do this first, BEFORE applying command line switches.

So the command line switches act like another control file.
In particular

flx --test=dir

current sets FLX_INSTALL_DIR and the cascade.

The problem is .. now the environment variables are irrelevant
because they're ALREADY been processed and overwritten,
not by setting a core layout directory, but by setting a cascade
variable.

that means, for example, that an attempt set the CACHE to somewhere
other than the default may fail if --test= cascades over it.
normally it won't, because the cache related to HOME and not the
installation point .. but a particular target CAN specify a target specific
cache .. if it does, environment variable will not override it
(because the env variable is applied before we even know the target).

the point is: the rules are so complicated I can't figure out how to do
it so it makes sense and leaves one in control .. other than demanding
each core variable be set and not allowing any default cascades.

that would be untenable on the command line of "flx": there are a LOT
of these variables to specify (about 15 I guess at the moment).

[Martin DeMello]

Since we're documenting stuff, it'd be helpful to list out everything that needs to be configured. My understanding thus far is

1. The felix installation:

    PREFIX/

        lib//felix/felix-latest = FLX_INSTALL_DIR

            host

            share

        bin

2. The cache:

    HOME/

        .felix

3. The target config:

    ???

The installation should be self-contained and read-only at runtime, and if there are several installations there should be a single config variable to select the one you want.

The target config should be a set of variables configurable when you compile a project. I can think of three things to configure:

- the felix installation

- the host architecture

- the c++ compiler and linker settings and flags

The cache depends on both the installation and the target config; I'd suggest putting it in .felix/<computed hash value>/ - the user should not have to handle any details of the cache location.

I would suggest having the target config be part of the project directory if it needs to be overridden - you could search upwards from `cwd` till you find a felix-config.fpc file and fall back on the default if it doesn't exist. I also see no problem with demanding that all the core variables be explicitly set if you have (a) a way to just run "flx" with no config at all and (b) a way to generate a default file (maybe just copy it from share/) and then let the user override what they need to. Case (a) takes care of new or casual users who want to explore felix without delving into fiddly details, and case (b) can be done once per project and then forgotten about, which is not too onerous if you're doing something large enough to need explicit configuration changes.

martin

[john skaller]

On 03/09/2014, at 3:54 AM, Martin DeMello wrote:

> Since we're documenting stuff, it'd be helpful to list out everything that needs to be configured.

During the build everything is automatic. There are no options.
The install follows "standard layout" and goes to /usr/local/lib/felix/felix-version
unless you change it like

sudo make PREFIX=/usr install

However you don't have to install anything. build/release is a perfectly good
Felix system.

Now, with the new system there are a list of about 20 variables that
control processing. You will be able to set them all individually
with a control file once I'm done.

> The installation should be self-contained and read-only at runtime,

SHARE and TARGET are both read-only, at the moment.
That's why I use a cache.

you can have any number of targets (for the SAME source version).
And any number of versions as well :)


BTW: SHARE is source code ONLY, and it is universal.

TARGET contains two things: configuration text for a particular setup,
such as #include files, fpc package descriptors, etc, and,
it contains binaries (object files, libraries, executables).

Everyone has to have at least one target, namely "host".

> The target config should be a set of variables configurable when you compile a project. I can think of three things to configure:
> - the felix installation
> - the host architecture
> - the c++ compiler and linker settings and flags
>
> The cache depends on both the installation and the target config; I'd suggest putting it in .felix/<computed hash value>/ - the user should not have to handle any details of the cache location.

Unfortunately there's a remaining problem, if you're doing parallel compilations.

Actually you don't need to use a hash, just use the target directory.
you end up with REALLY long filenames though :-)
That's because the file *within* the cache are absolute filenames.
So you'd end up with /usr/....cache/usr/....target/usr....file ... :)

>
> I would suggest having the target config be part of the project directory if it needs to be overridden - you could search upwards from `cwd` till you find a felix-config.fpc file and fall back on the default if it doesn't exist. I also see no problem with demanding that all the core variables be explicitly set if you have (a) a way to just run "flx" with no config at all and (b) a way to generate a default file (maybe just copy it from share/) and then let the user override what they need to. Case (a) takes care of new or casual users who want to explore felix without delving into fiddly details, and case (b) can be done once per project and then forgotten about, which is not too onerous if you're doing something large enough to need explicit configuration changes.

unfortunately there are many use cases. Here's one, and a real one:

My friend RF does iOS apps. It turns out .. there are FIVE different
targets required to build for iOS and get coverage.

So when you're building .. you actually want to build all five from
a SINGLE project :)

in any case, what you're describing is basically correct except it only
describes the core system. to allow "add on packages" we need to
provide search paths.

At present the addons should not be version specific, so the addon
directory is yet another directory to specify.

[Martin DeMello]

On Tue, Sep 2, 2014 at 11:35 AM, john skaller <ska...@users.sourceforge.net> wrote:

My friend RF does iOS apps.  It turns out .. there are FIVE different
targets required to build for iOS and get coverage.

So when you're building .. you actually want to build all five from
a SINGLE project :)

Could this still be done in one config file with [host = ...] sections, so that you still need to pass a single config flag to select one of them?

 

in any case, what you're describing is basically correct except it only
describes the core system. to allow "add on packages" we need to
provide search paths.

At present the addons should not be version specific, so the addon
directory is yet another directory to specify.

Can you describe the package requirements a bit more? I am still hopeful that 0install can be made to work for this.

martin 

[john skaller]

On 03/09/2014, at 4:51 AM, Martin DeMello wrote:

> On Tue, Sep 2, 2014 at 11:35 AM, john skaller <ska...@users.sourceforge.net> wrote:
>
> My friend RF does iOS apps. It turns out .. there are FIVE different
> targets required to build for iOS and get coverage.
>
> So when you're building .. you actually want to build all five from
> a SINGLE project :)
>
> Could this still be done in one config file with [host = ...] sections, so that you still need to pass a single config flag to select one of them?

I don't know. Certainly not initially. However the objective is that:

flx --felix=whatever.fpc .,...

will provide most of the things that vary from project to project in a single
switch. And perhaps we can make

flx-p ....

work so -p looks for prohect.fpc in the current directory.

> Can you describe the package requirements a bit more? I am still hopeful that 0install can be made to work for this.

I don't entirely know. Here are two classes of package:

(a) Felix only core package (goes in SHARE)
(b) Felix only user package (goes in PACKAGES)

Both these packages are installed *entirely* by copying/unpacking,
and will probably contain text data and Felix code.

The first kind is version specific, and probably what you would
call "mandatory". It will allow splitting up the Felix system a bit.,

The second kind is "less core" stuff.

Now, consider Felix code doing C++ bindings. The code
is universal, but to compile or link you need to know how
to find #includes and objects/libraries. You do that with
an *.fpc file, but such a file is TARGET specific, and possibly
platform dependent. (meaning the user has to tell the system
where to find it).

As an example, SDL bindings.

Then we get the next level: stuff including C/C++ code
requiring compilation. This may actually be easier than the
above, because Felix already knows how to compile C++,
and so it can decude where in the TARGET to put the result.
So the *.fpc file can be generated.

So for this one, the package manager has to do some actual
work .. at the moment.

I plan for autocompile of C++ which would eliminate that,
but it isn't here yet.

Ok so now the issue is: what does 0install actually do?
Apart from being long winded and stupidly starting words
with a digit .. What assumptions does it make?
What are the constraints?

Yo know it says on the home page "you don't have to guess"
but I found myself doing a lot of guessing about where it
put stuff. It says "supports both binary and source packages" which
immediately raises suspicions.

And of course its a third party dependency.

How is it going to manage multiple targets of Felix
systems? Etc.

The question is how much work is it to do it in 0install
compared to doing it in Felix itself?

Since IMHO most of the work has to be done in Felix by
Felix for Felix anyhow.

BTW: we could of course make a 0install package for Felix itself.
But the question is: why? Git clone, make build is fairly simple.

[john skaller]

> The question is how much work is it to do it in 0install
> compared to doing it in Felix itself?

So do you want to see "Scary"??

http://zero-install.sourceforge.net/install-osx.html

When I see that I'm not even going to bother trying :-)
Neither will anyone else. GTK? You're kidding!

Its bad enough getting SDL to build.

[Martin DeMello]

gtk is purely optional, and just used for the gui. where i'm coming from is this:

1. if you have packages, you ideally want a package manager

2. the package manager will need to do a bunch of things that aren't really felix-related

3. so why not use the existing work that the 0install guys are doing, especially since one of their stated goals is to be used for language packages

also, the longer i work in software, the more suspicious i am of not-invented-here syndrome :)

to answer your question about what 0install would actually do, i picture the following scenario:

$ 0install felix-sdl-bindings

# installs the bindings in some standard location

# invokes the felix compiler as needed to finalise the installation

$ flx --pkg=sdl-bindings myproject.flx

# finds the sdl-bindings package via 0install, and sets up the compile and link paths accordingly

# builds myproject

$ 0install felix-myproject

# looks up the 0install package, finds felix-sdl-bindings as a dependency

# builds everything

it would essentially allow a similar workflow to opam + ocamlfind, which i find works very nicely for a developer-workflow-friendly packaging solution.

martin

[Ryan]

I got SDL builds right the first time.

I think you just have really, really, REALLY bad luck building stuff.

[john skaller]

On 03/09/2014, at 10:19 AM, Martin DeMello wrote:

> gtk is purely optional, and just used for the gui.

I know .. I actually do have 0install.
(the older Python version).

Note: nothing stops anyone using 0install with Felix
or anything else.

> where i'm coming from is this:
>
> 1. if you have packages, you ideally want a package manager
> 2. the package manager will need to do a bunch of things that aren't really felix-related

Dowloading and maintaining packages, yes.

> 3. so why not use the existing work that the 0install guys are doing, especially since one of their stated goals is to be used for language packages

There are several reasons. Similar to ReadTheDocs. Third party dependence
is one of the reasons. Felix bundles Judy, Google RE2, Sqlite3, OCS Scheme,
and Dypgen for good reason.

The Felix build programs are replacing fbuild, which is bundled and written
in Python (and very good), for a good reason too. The author isn't on
the project any more and I can't maintain it.

> also, the longer i work in software, the more suspicious i am of not-invented-here syndrome :)

The more I work on it .. and it's over 40 years now .. the more suspicious I am (period). :)

0install provides a way to run a *program* using a URL. Felix isn't really a program.

>
> to answer your question about what 0install would actually do, i picture the following scenario:
>
> $ 0install felix-sdl-bindings

You have to give a URL I think:

0install http://felix-lang.org/packages/felix-sdl-bindings

> # installs the bindings in some standard location

How does it discover where that is?

> # invokes the felix compiler as needed to finalise the installation

Right so how is this much different from

curl http://felix-lang.org/packages/felix-sdl-bindings.pak
sudo flx_iscr --outdir=$FLX_PACKAGE_DIR felix-sdl-bindings.pak
sudo flx --felix=package-admin.fpc $FLX_PACKAGE_DIR/builder.flx --target=$FLX_TARGET_DIR

or something similar?

APART from the signature verification.

Its true, if the process above had dependencies, they wouldn't be handled.

However, it is not so easy to handle them. Here's why:

You think that to install the GUI you need flx_sdl which needs SDL.
Right?

WRONG. You don't need SDL at all to install sdl-bindings.
You only need SDL if you want to compile the generated C++.

My point: Felix has two levels of dependencies. Most package
managers have tricky ways that could handle this but then
you're programming using tricks to do it.

Why don't you try it? There are several packages you could make:

1) Gnu GMP .. I think there's a binding already.

2) Sqlite3 .. its in the bundle but it isn't used for anything

3) Regexps .. as you know Felix has a user space grammar
so there are three parts to this:

(a) the RE2 bindings
(b) a combinator library provides regular terms
(c) a syntax for regdefs

4) SDL

5) Any other things?

At least one issue with Felix is that installing universal sources
and building platform dependent binaries are two different things.
Because there is no "the architecture". There can be any number
of targets.

How do you know which ones to build and when?
What happens if you upgrade Felix?

[john skaller]

On 03/09/2014, at 12:02 PM, Ryan wrote:

> I got SDL builds right the first time.
>
> I think you just have really, really, REALLY bad luck building stuff.

I have a 7 year old Macbook Pro, a broken clang I cant
upgrade, an old gcc Apple won't upgrade and I'm running
OSX 10.6.8 which I can't upgrade and no one supports any more.

I also have flakey wireless only internet and not much money
to pay for it.

Yes, there's a clang binary for OSX. .. OSX Mavericks that is.
It doesn't work on Snow Leopard.

If you're running on a fast Linux box, building stuff like GTK is easier
because it was developed for Linux with Linux tools etc etc.

[john skaller]

Felix now has this:

class Config {
typedef config_type = (
FLX_SHARE_DIR: string,
FLX_TARGET_DIR: string,
FLX_HOME_DIR: string,
FLX_PROFILE_DIR: string,
FLX_CONFIG_DIR: string,
FLX_CACHE_DIR: string,
FLX_OUTPUT_DIR: string
);

SHARE: directory containing shared universal resources "text only"
[white lie: can also contain *.jpg etc for website]

TARGET: directory containing target specific resources including binaries

HOME: usually $HOME/.felix, top level of directory Felix can get
personal details from, and which it may write to

PROFILE: usually $HOME/.felix/config, the directory containing
personal package control files (*.fpc)

CONFIG: directory containing target specific package control
files

CACHE: place to read and write cached binaries

OUTPUT: place to put generated code, usually C++

This list isn't complete. Note the pseudo variables FLX_INSTALL_DIR
and FLX_TARGET_SUBDIR are not included.

All of these variables can be set by and

(a) the environment
(b) a control file

and some can be set by

(c) command line switches.

They can be set individually, as well as FLX_INSTALL_DIR
and FLX_TARGET_SUBDIR, and HOME. However FLX_TARGET_SUBDIR
may not be set in a control file without previously setting FLX_INSTALL_DIR.
This is because there is no record of FLX_INSTALL_DIR to cascade
the setting of FLX_TARGET_DIR from.

When a variable is set, dependent variables are also reset.
So the order of setting matters.

FLX_INSTALL_DIR cascades to set FLX_SHARE_DIR and FLX_TARGET_DIR
for example.

HOME cascades to FLX_HOME_DIR and thence FLX_PROFILE_DIR.

The config control variable then cascade to other control variables,
such as the name of the grammar file, the search path for
libraries, etc. Most of my recent hacking has been getting this
cascade correct.

Here is a control file I'm using:

~/felix>cat build.fpc
FLX_INSTALL_DIR: build/release
FLX_CACHE_DIR: build/release/cache/binary
FLX_OUTPUT_DIR: build/release/cache/text

with command:

build/release/host/bin/flx --felix=build.fpc hello

More variables will be added to the config,
This is your "one stop shop" for controlling a project.
It makes all the compiles and whatever in that project
build in the same environment with a "one point"
control.

Felix looks in FLX_PROFILE_DIR for a file named "felix.fpc"
and if found processes it as a config file. It also looks for this
in the FLX_TARGET_DIR/config directory.

note that environment variables are OVERRIDDEN by a
control file. The order is:

defaults
environment
control file
command line switches

I am thinking to add a second set of environment variables such as:

FLX2_CACHE_DIR

These will override the command line switches. The reason is
if you have script, eg in a makefile, and you want to modify
the behaviour: at the moment that is invasive. This is particularly
bad if the script is Felix not bash script (because it triggers recompiles
in the code being used to compile things .. )

==================================================

Although it sounds simple there are some NASTIES floating about.
Consider this, which is actually done more or less:

build/release/host/bin/flx build/release/src/tools/flx.flx hello

Yep, that's flx building itself. The problem? The environment
controlling the first "flx" is the same as the second one.
They'll use the same cache for example.

This is a real problem for some other variables.
LD_LIBRARY_PATH is the worst.

If you have A loading B loading C, then A and B use
the SAME value of LD_LIBRARY_PATH. There's nothing
you can do about it. The only way to fix this is run C
as a separate process because that's the only way you can
change LD_LIBRARY_PATH (using setenv() in your
program code does NOT work).

[john skaller]

I have now implemented stuff which enables part
of the package management system. It works like this:

1. Put Felix code in a nice_place.
2. Add

FLX_LIB_DIRS: nice_place

to your layout control file. The default one is:

$HOME/.felix/config/felix.fpc

The Felix compiler flxg will now search for include files
in directory "nice_place" as well as the usual places.

That's it!

The recommended nice place is:

/usr/local/lib/felix/packages/share/lib

You can have any number of nice places,
and you can include spaces in filenames with quotes:

FLX_LIB_DIRS: nice_place "another nice place"

(and quotes by using the other kind of quote .. escapes
are not supported because the escape char \ is also
the separator on Windows).

(at least .. i think that will work :)

How you get stuff into your package directory is up to you.

This mechanism will work now for pure Felix code.

For bindings to C/C++: in order to support #include
file searching, static linkage,load time dynamic linkage
and run time dynamic linkage, Felix usually uses
a package directive like:

.. requires package "gmp" ...

which in turn requires a file

gmp.fpc

to exist in the package database. That is normally found in:

$FLX_CONFIG_DIR

which is usually

$FLX_TARGET_DIR/config

which is usually

FLX_INSTALL_DIR/host/config

In other words, we need gmp.fpc to be in the target
config directory, but this is tricky. Such a file would
be clobbered on an upgrade. I do not know how
to do this yet. The package file is going to be different
for each target, in fact it may be gmp isn't available
on say iPhone.

Probably we will have

FLX_PACKAGE_DIR/target/config

to put the files in (for each target). The user will have to do this.
Felix can't know how to find things or what name various libraries
have. Some kind of guessing system might be useful.
Packages might come with prebuilt configs for popular layouts
(eg Debian Linux, -lgmp -lgmpxx works I think).

This means the next step will be to allow the user to
specify extra config directories to search.

Note of course if you do not use packages in your package .. er
I mean you do not use "require package" then you can just
manually specify the switches on the flx command line.

[john skaller]

I think I am going to change how --target works.
At present, it selects a subdirectory of FLX_INSTALL_DIR
and makes that FLX_TARGET_DIR.

Instead, you should use

--target=mytarget

will just be a short hand for

--felix=FLX_PROFILE_DIR/config/mytarget.fpc

In other words, it has nothing to do with the target directory,
it just selects a control file in your profile directory.

[Recall FLX_PROFILE_DIR defaults to $HOME/.felix]

This make sense because mytarget.fpc can set the FLX_TARGET_DIR
itself, but also it can set other things like search paths or whatever
(at least it will be able to when i finish with it :)

[Martin DeMello]

That seems like a nice, clean design!

martin

[john skaller]

So here is my tests case:

I start with this file:

//========================================
// tgmp.flx
//========================================
include "gmp";
open Gmp;

var x : mpz = mpz_of_int 42;
var y = x + x;
println$ y;
//=======================================

and I want to do this:

flx --felix=build.fpc tgmp

and have it work.

First:

sudo mkdir /usr/local/lib/felix/packages
sudo mkdir /usr/local/lib/felix/packages/share
sudo mkdir /usr/local/lib/felix/packages/share/lib
sudo mkdir /usr/local/lib/felix/packages/host
sudo mkdir /usr/local/lib/felix/packages/host/config

I make the package directory. I have chosen to make
it "in the image" of the usual install directory.

Next I get this file:

https://raw.githubusercontent.com/felix-lang/gmp/master/GMP/gmp.flx

(I just used my browser) and copy it into the package directory

sudo cp gmp.flx /usr/local/lib/felix/packages/share/lib

So now, I need to make a config file for it called gmp.fpc

Name: gmp
Description: GNU Multiple precision Arithmetic
provides_dlib: -lgmpxx
provides_slib: -lgmpxx
requires_slibs: -lgmp
requires_dlibs: -lgmp

which I put in

/usr/local/lib/felix/packages/host/config

This works for me: gmp.h and gmpxx.h are in /usr/local/include
and libgmp.so and libgmpxx.so and libgmp.a and libgmpxx.a are
in /usr/local/lib, for some reason I don't understand these are
already searched by C__.

Note there is a BUG in the build of libgmpxx.so, it isn't linked to
libgmp.so as it should be probably because I copied it out
of /opt/local/lib. So I have to put in a requirement for a dlib,
this normally should never be required (the linker should
handle dependencies for dynamic libraries).

Now my control files is called build.fpc and looks like this:

FLX_INSTALL_DIR: build/release
FLX_CACHE_DIR: build/release/cache/binary
FLX_OUTPUT_DIR: build/release/cache/text

FLX_LIB_DIRS: build/release/share/lib /usr/local/lib/felix/packages/share/lib
FLX_CONFIG_DIRS build/release/host/config /usr/local/lib/felix/packages/host/config

So I'm running off build/release. However I'm searching the package
directory for Felix libs to find the binding we just installed,
and both the build/release host config database as well as the
package one.

Note if you set FLX_LIB_DIRS or FLX_CONFIG_DIRS you have to
set the complete list of them, the default paths are wiped out.

[john skaller]

OK, so now we have to get to some packaging rules.

A simple pure felix library package can simply be installed
somewhere in the library path. But how do we know what it is?

A core problem is: if we need an fpc file, how do we make it?

[Martin DeMello]

would something along the lines of package_path = pure_function(config, package_name, version) work? or is there a platform-dependent component to them too?

martin

[john skaller]

On 05/09/2014, at 11:36 AM, Martin DeMello wrote:

> would something along the lines of package_path = pure_function(config, package_name, version) work? or is there a platform-dependent component to them too?

A pure package doesn't need any config, that's the point of it.

For example, the String class, although partly a binding to C++
string type, doesn't need any config, because include file
<string> can be yanked in without a package: its ISO Standard
which we assume.

On the other hand an SDL binding requires SDL, freetype,
OpenGL, and half a dozen other things and there's no standard
reliable way to find where these are, or even what's needed,
for example on Linux you need libjpg for SDL_image to work,
on OSX you don't because Apple provides jpg natively.

Then there are packages that include C++ code.
For example some of the Felix RTL could be made into
packages, eg libJudy or sqlite3. Judy needs almost no
config. sqlite3 doesn't need any at all. However these things
have to be compiled into libraries and then we have to figure
out where to put the result.

At present, binaries go into the target/lib/rtl directory
or the target/bin directory on Windows if its a DLL
(WIndows uses PATH for DLLS).

Since we have toolchain things and core configuration done,
we can actually build C/C++ code in a standard way,
without configuration (modulo needing to know where
dependent headers and shared libs needed for linking are).

In scoop, Mike Maul's package manager, each package has some
executable code called setup() or something which is run to build
the package (its Felix code obviously).

That's all very well but such a function has to work on all platforms
for all targets (assuming the required support is available).

Otherwise we have to go to an interactive configuration, or, we
have to just leave it up to the user to write the config stuff.
In the later case, we would like the package to at least provide
some test cases to check stuff is working a bit (not exhaustive
but enough for *something* to work :)

Anyhow, at the moment the first step would seem to be to get "flx"
to be able to use add on packages that are retained when the
Felix core is rebuilt: at least the sources should remain.

The real trick is to auto-compile C++ so it happens behind the scenes
on demand.

This doesn't solve the config problem, but it solves a lot of problems
related to building (roughly: forget it, its automatic).

It isn't unreasonable to also make a system that can generate
config files after searching using hints from the package and
a platform specific search method. For example Debian
systems put most apt-get installed packages in /usr/lib+/usr/include
and the library FRED is called libFRED.so libFRED.a etc.

[john skaller]

I actually have this problem now in flx so I thought I'd solve it
using fibres instead of callbacks. Here's a comparison
of a simplified example (not tested!).

//===================================================
// Problem: Messages type A processed by routine A.
// Add type B, routine B.
// Errors for none of the above, type E.

union msg = mA | mB | mE;

//===================================================
// Solution 1: callbacks/subroutines
//
// Concept: process what we can, if we can't
// pass the buck.

// Separated components
proc A (nxt: msg->0) (m:msg) {
match m with
| mA => println "A";
| ?x => nxt x;
endmatch;
}

proc B (nxt: msg->0) (m:msg) {
match m with
| mB => println "B";
| ?x => nxt x;
endmatch;
}

proc E (m:msg) {
match m with
| mE => println "Error";
| => assert false;
endmatch;
}


// Integrator
fun processor () : msg -> 0 = {
var clsB = B E;
var clsA = A clsB;
return clsA;
}

// Controller
proc controller (msgs: list[msg]) {
var p = processor ();
for m in msgs do p m; done
}

//=========================================================
// Solution 2: fibres/coroutines
//
// Concept: process what we can, if we can't
// pass the buck.

proc fA (nxt: oschannel[msg]) (m:ischannel[msg]) () {
while true do
match read m with
| mA => println "A";
| ?x => write nxt,m;
endmatch;
done
}

proc fB (nxt: oschannel[msg]) (m:ischannel[msg]) () {
while true do
match read m with
| mB => println "B";
| ?x => write nxt,m;
endmatch;
done
}

proc fE (m:ischannel[msg]) () {
while true do
match read m with
| mE => println "Error";
| _ => assert false;
endmatch;
done
}

// Integrator
proc fprocessor (o: &oschannel[msg]) {
var iE,oE = mk_ioschannel_pair[msg];
var pE = fE iE;
var iB,oB = mk_ioschannel_pair[msg];
var pB = fB oE iB;
var iA,oA = mk_ioschannel_paior[msg];
var pA = fA oB,iA;
spawn_fthread pE;
spawn_fthread pB;
spawn_fthread pA;
o <- oA;
}

// Controller
proc fcontroller (msgs: list[msg]) {
var o : oschannel[msg];
fprocessor (&o);
for m in msgs do write$ o,m; done
}
//==============================================================

As you can see the fibre solution is a tad longer.
So what's the advantage?

In this simplified case there is none!

The callback based solution is perfectly good...
.. or is it?

No, it isn't. Its nowhere near as good. Here's why:

SPECIFICATIONS CHANGE.

Suppose now our processing of a message
depended on the last one, in other words
there's an ordering dependence. This is true in
the real case I'm dealing with (namely,
if you set FLX_TARGET_SUBDIR you have to have
previously set FLX_INSTALL_DIR because its a subdirectory
of the install directory).

Now we need state to do it right. The callback solution
as written has nowhere to put the state!

Of source we can fix this:

fun make_callback () {
var state: state_t;
proc callback (m:msg) { .. if state == ... }
return callback;
}

However, now we have to write
a switch on the state, to handle the message correctly, since the action
is state dependent.

On the other hand with the fibre we could just add the state variable
as a local variable. But actually there's a better way:

USE THE PROGRAM COUNTER AS THE STATE VARIABLE.

That way we need no explicit variable, no type for it, and no switch!

Note: this is, in fact, equivalent to the functional programming method
of continuation passing (because the program counter IS the continuation).

Consider for example message types A1 and A2:

proc fA (nxt: oschannel[msg]) (m:ischannel[msg]) () {
A1:>>
match read m with
| mA1 => println "A1"; goto A2;
| ?x => write nxt,m; goto A1;
endmatch;
A2:>>
match read m with
| mA2 => println "A2"; goto A1;
| ?x => write nxt,m; goto A1;
endmatch;
}

Notice out of order message are passed on in this particular
solution and will be trigger an assert false in fE.

Of course this is a finite state machine with two states, and the
callback based solution would be similar except:

1) Instead of using the "stack and program counter" to maintain
state, it just uses a variable, external to the callback.

2) It has to put the variable INTO the program counter with a switch
to go to the right handler code.

Although this is not too onerous with a finite state machine with two
states .. if your logic is a push down automata with a more complex
state you will end up EMULATING recursion and a stack.

With the fibre solution.. well the implementation is the same.
The difference is Felix does the emulation for you.
It actually generates callbacks with a switch.

In really complex situations you need manually maintained
state variables, however the Felix mechanism reduces the
amount of state you have to manage (to whatever you cannot
fit into a notional stack/program, counter).

Only .. even that is not true. Why? Because, when it gets complex
you can always add an extra stack/program counter. How?

Well you launch another fibre of course!!