HashState modulo

[Ralf Hemmecke]

Hi Waldek,

I just stubled over the line.

modulo(hs : %, i : I) : I == (HASHSTATEMOD(hs, i)$Lisp) pretend I

https://github.com/fricas/fricas/blob/master/src/algebra/hashstate.spad#L73

Could be removed, since it is not exported and nowhere used since we use
hashUpdate! for every domain of category SetCategory.

Ralf

[Waldek Hebisch]

If you want to remove it just go on.

I think that we should change how hashing works (I probaly already
wrote something about it). Currently SetCategory unconditionally
exorts 'hash' and 'hashUpdate!'. However, for many domains I see
no way to implement reasonable 'hashUpdate!', that is one giving
apporximately random distribution. I think we should have
separate category Hashable exporting 'hash' and 'hashUpdate!'.
And domains should assert Hashable only when they actually
implement useful 'hashUpdate!'.

--
Waldek Hebisch

[Ralf Hemmecke]

> If you want to remove it just go on.

Will do soon.

> I think that we should change how hashing works (I probaly already
> wrote something about it). Currently SetCategory unconditionally
> exorts 'hash' and 'hashUpdate!'. However, for many domains I see
> no way to implement reasonable 'hashUpdate!', that is one giving
> apporximately random distribution. I think we should have
> separate category Hashable exporting 'hash' and 'hashUpdate!'.
> And domains should assert Hashable only when they actually
> implement useful 'hashUpdate!'.

Oh, I am not at all against that, rather I think Hashable should be the
better way of doing it.
I guess you want to release soon (when approximately?). Then maybe we
do that with HashTable afterwards. I would rather like this hunchentoot
thing inside FRICASsys as I described here

https://hemmecke.github.io/fricas/install.html#jfricas-optional

patches are in the wip/fricas-jfricas branch of my github repo.
The above basically describes a way to build everything from the gitrepo
that could then make a distribution.

In fact, I would also like to put the new install.rst into the repo.
Maybe more on this in a separate "release thread".

Ralf

[Waldek Hebisch]

On Wed, May 03, 2023 at 09:31:00PM +0200, Ralf Hemmecke wrote:
> I guess you want to release soon (when approximately?).

Good time would be June, but there really a lot of things which
should be done first.

> Then maybe we do
> that with HashTable afterwards. I would rather like this hunchentoot thing
> inside FRICASsys as I described here
>
> https://hemmecke.github.io/fricas/install.html#jfricas-optional
>
> patches are in the wip/fricas-jfricas branch of my github repo.

Hmm, in one place you mention wip/fricas-sbcl-hunchentoot, which
one should I look at.

> The above basically describes a way to build everything from the gitrepo
> that could then make a distribution.

Well, quicklisp dependence is very problematic. One trouble is that
during release build there is no network access. Another is that
it seem to be unversioned, so there is no way to say "works with
P1 version x1.y1, P2 version x2.y2, ..., Pn verion xn.yn". It
seems that to work around this we will need to bundle everything
that quicklisp fetches from the net.

Another trouble is hardcoded paths. FriCAS is supposed to find
all files that it needs relative to path specified by FRICAS
environment variable. Currently for Lisp files I use simple
approach: everything that is needed is loaded and dumped as part
of core. I am not sure if this will work for hunchentoot,
but that would be my first try,

--
Waldek Hebisch

[Ralf Hemmecke]

>> Then maybe we do that with HashTable afterwards. I would rather
>> like this hunchentoot thing inside FRICASsys as I described here
>>
>> https://hemmecke.github.io/fricas/install.html#jfricas-optional
>>
>> patches are in the wip/fricas-jfricas branch of my github repo.
>
> Hmm, in one place you mention wip/fricas-sbcl-hunchentoot, which one

> should I look at.aaOoop...

I say

git clone -b wip/fricas-sbcl-hunchentoot
https://github.com/hemmecke/fricas.git

on the above URL (install.html), then it is certainly that.
I locally work on wip/fricas-jfricas but never pushed that back to
github. Anyway wip/fricas-sbcl-hunchentoot is the one.

>> The above basically describes a way to build everything from the
>> gitrepo that could then make a distribution.

> Well, quicklisp dependence is very problematic. One trouble is that
> during release build there is no network access.

OK, but you must somehow get all fricas prerequisites onto that machine.
I experimented with Kurt and using quicklisp was the easiest to get
hunchentoot into a loadable situation without juggling too much with a
lot of problems.

There are two problems to solve.

1) easily build from github sources (getting prerequisites on the fly)
for end users that build fricas themselves

2) build a distribution as a release manager

I was not aware that you insist on "no network" during release build
(but actually, at the time you are able to start "configure && make" no
network is concacted).

I think the whole thing can also made to work with an apt installed
hunchentoot, but the boilerplate code would be too complicated to care
for every platform.

You can also install a quicklisp environment with hunchentoot already
loaded (just as any other prerequisite). That is why on install.html all
that stuff is listed in the prerequisites section. If you are in the
section "Detailed installation instructions", then network access is
only needed to get the fricas sources.

> Another is that it seem to be unversioned, so there is no way to say
> "works with P1 version x1.y1, P2 version x2.y2, ..., Pn verion
> xn.yn". It seems that to work around this we will need to bundle
> everything that quicklisp fetches from the net.

OK, that seems indeed to be a problem, not being able if quicklisp
doesn't let me pin some versions of the things I fetch. I must look into it.

> Another trouble is hardcoded paths. FriCAS is supposed to find all
> files that it needs relative to path specified by FRICAS environment
> variable. Currently for Lisp files I use simple approach:
> everything that is needed is loaded and dumped as part of core.

If you saw some paths on the above URL, then they are deliberately done
through environment variables. Nothing is hardcoded. I just used FDIR as
a common place to do all the work. When that directory is remove,
nothing else on your computer would be change. (That was done for users
that want to install themselves, but is actually also interesting for
building a distribution.

The other thing is, once FRICASsys is built, and put into the binary
release tarball, then the target system (end user) does not have to have
quicklisp or hunchentoot, because hunchentoot is inside the FRICASsys
image. That's the whole idea -- hunchentoot into the image. There is
then also no need to fiddle around with SBCL_HOME.

I'm a bit busy with other things, but will try to open a pull request on
github so that we can discus further on the PR and hopefully Kurt will
then also chime in and comment.

I hope that clarifies the big picture.

Ralf

[Waldek Hebisch]

On Thu, May 04, 2023 at 12:46:15AM +0200, Ralf Hemmecke wrote:
> > > Then maybe we do that with HashTable afterwards. I would rather like
> > > this hunchentoot thing inside FRICASsys as I described here
> > >
> > > https://hemmecke.github.io/fricas/install.html#jfricas-optional
> > >
> > > patches are in the wip/fricas-jfricas branch of my github repo.
> >
> > Hmm, in one place you mention wip/fricas-sbcl-hunchentoot, which one
> > should I look at.aaOoop...
>
> I say
>
> git clone -b wip/fricas-sbcl-hunchentoot
> https://github.com/hemmecke/fricas.git
>
> on the above URL (install.html), then it is certainly that.
> I locally work on wip/fricas-jfricas but never pushed that back to
> github. Anyway wip/fricas-sbcl-hunchentoot is the one.
>
> > > The above basically describes a way to build everything from the
> > > gitrepo that could then make a distribution.
>
> > Well, quicklisp dependence is very problematic. One trouble is that
> > during release build there is no network access.
>
> OK, but you must somehow get all fricas prerequisites onto that machine.
> I experimented with Kurt and using quicklisp was the easiest to get
> hunchentoot into a loadable situation without juggling too much with a
> lot of problems.
>
> There are two problems to solve.
>
> 1) easily build from github sources (getting prerequisites on the fly)
> for end users that build fricas themselves

If the user wants quicklisp I have nothing againt it as a possiblity.
OTOH I would like users to be able to fetch, disconnect and build.
Another thing is ability to recreate old versions.

> 2) build a distribution as a release manager
>
> I was not aware that you insist on "no network" during release build
> (but actually, at the time you are able to start "configure && make" no
> network is concacted).

I am not sure what is current practice of Linux distributions, but
past standard was that all files were fetched by distribution
infrastructure. Network access during build was forbidden. So
"no network" during build was/is important for compatibility
with packaging FriCAS for Linux distributions.

If you want reasons:
- security of build infrastructure: with explicit list of sources
trust is explicit (easy to inspect what is trusted) and limited.
With fetching during build trust is distibuted and hard to inspect.
- reproduciblity and verification: package descriptions contain
cryptographic checksums, so there is warranty that for given
package version the sources will stay constant
- people in many part of world have intermitent net access, they
can fetch sources, but only at some times/places

> > Another trouble is hardcoded paths. FriCAS is supposed to find all
> > files that it needs relative to path specified by FRICAS environment
> > variable. Currently for Lisp files I use simple approach:
> > everything that is needed is loaded and dumped as part of core.
>
> If you saw some paths on the above URL, then they are deliberately done
> through environment variables. Nothing is hardcoded. I just used FDIR as a
> common place to do all the work. When that directory is remove, nothing else
> on your computer would be change. (That was done for users that want to
> install themselves, but is actually also interesting for building a
> distribution.
>
> The other thing is, once FRICASsys is built, and put into the binary release
> tarball, then the target system (end user) does not have to have quicklisp
> or hunchentoot, because hunchentoot is inside the FRICASsys image. That's
> the whole idea -- hunchentoot into the image. There is then also no need to
> fiddle around with SBCL_HOME.

OK, I will how this works.

> I'm a bit busy with other things, but will try to open a pull request on
> github so that we can discus further on the PR and hopefully Kurt will then
> also chime in and comment.

You gave me above branch to fetch, that is enough for me, pull request
does not make it easier (at least for me).

--
Waldek Hebisch

[Waldek Hebisch]

On Wed, May 03, 2023 at 09:31:00PM +0200, Ralf Hemmecke wrote:

> Then maybe we do
> that with HashTable afterwards. I would rather like this hunchentoot thing
> inside FRICASsys as I described here
>
> https://hemmecke.github.io/fricas/install.html#jfricas-optional
>
> patches are in the wip/fricas-jfricas branch of my github repo.

I think that it is worth mentioning that one of my significant
troubles working with FriCAS (and in general) is managing work
queue. Simply "getting up to speed" takes time and if something
is suspended I need time just to restart it. IME most efficient
strategy is to do something in one shot. But things may get
blocked and work lands on work queue. Also, I usually can
allocate to FriCAS only relatively short uninterruped time
intervals. So I instead of working on "highest priority"
item I frequently do smaller things which can be done in
available time. Trying to "work on highest priority item",
in many cases would mean not making any progress at all.

Coming back to hunchentoot/jfricas: I consider this a high
priority item. But as I wrote in another mail (and in mails
long ago) for release there are issues to resolve. One
of issues (dependency on SSL) is resolved. But there are
other, and progress here requires cooperation. Which brings
communication delays. So this is definitely not thing that
cound be "fast tracked".

--
Waldek Hebisch

[Tim Daly]

Waldek,

As I'm sure you've come to appreciate it takes a lot of time to understand,

make, test, and publish solutions to what may seem to others as "simple

problems".

There was a huge discussion about a release schedule for Axiom. Some

wanted a weekly release. We eventually agreed on a release every 2 months.

Fricas seems to have much longer intervals with no defined schedule.

People seem to forget that you have, like I had, "a real job" that likely

does not involve Fricas or their particular problem. Axiom work was

always a "nights and weekend" effort.

It is easy to find a problem. It is much harder to find a solution.

Ask people to propose a patch rather than inventing it yourself.

Otherwise the process doesn't scale.

Tim

[Waldek Hebisch]

I have now preliminary way to build sbcl+hunchentoot from a tarball.
It requires no network access during build and hopefully all
trash is limited to build directory (which can be removed just
after build).

To try:

wget http://www.math.uni.wroc.pl/~hebisch/fricas/hsbcl.tar
tar -xf hsbcl.tar
cd hsbcl
time ./build_hsbcl > rap1 2>&1

this creates executable called 'hsbcl' which can be used to
build FriCAS.

If that is enough for jfricas, then I will do something similar for
next release (but probably better integrated with FriCAS build).

--
Waldek Hebisch

[Ralf Hemmecke]

>> There are two problems to solve.
>>
>> 1) easily build from github sources (getting prerequisites on the fly)
>> for end users that build fricas themselves
>
> If the user wants quicklisp I have nothing againt it as a possiblity.
> OTOH I would like users to be able to fetch, disconnect and build.
> Another thing is ability to recreate old versions.

I have seen (but not yet tested) your hsbcl stuff.

http://www.math.uni.wroc.pl/~hebisch/fricas/hsbcl.tar

What I like is that you basically give explicit version numbers in the
parts that are necessary to build hunchentoot. Yes, I support the idea
of having concrete version numbers.

Where I would be not so happy about is to put those zip (or source
files) into the fricas repository. I would accept to put it into the
distribution tarball, but that is only half of the story. A user should
also be able to build from the git sources. So it would be good to have
a script that can be run before "configure" in order to get those
additional prerequisites.

Since jfricas is currently restricted to sbcl and FriCAS can be
considered a something on top of quite a variety of lisps, I am actually
not even sure whether it is right to put the hunchtoot stuff into a
source distribution. In the binary (on top of sbcl) distribution, it
would certainly make sense.

>> 2) build a distribution as a release manager

> I am not sure what is current practice of Linux distributions, but
> past standard was that all files were fetched by distribution
> infrastructure. Network access during build was forbidden. So
> "no network" during build was/is important for compatibility
> with packaging FriCAS for Linux distributions.

I have no problem with the no-network-paradigm. The only question is how
to tell the user (via documentation or via a little script) how to get
the prerequisites if "configure --enable-webspad" (i.e. jfricas) is
wished. A user would anyway have to install jupyter himself.

> To try:
>
> wgethttp://www.math.uni.wroc.pl/~hebisch/fricas/hsbcl.tar

> tar -xf hsbcl.tar
> cd hsbcl
> time ./build_hsbcl > rap1 2>&1
>
> this creates executable called 'hsbcl' which can be used to
> build FriCAS.

I've build fricas with hsbcl and no quicklisp and installed jfricas on
top of this. The whole thing works with hsbcl and the current master.
No additional patches from my side are required. In other words, if a
user provides a version of SBCL that lets webspad start hunchentoot

https://github.com/fricas/jfricas/blob/master/jfricas/webspad.lisp#L217

then jfricas can be installed roughly as described at
https://hemmecke.github.io/fricas/install.html#jfricas-optional
The part with quicklisp and hunchentoot installation can be skipped.

I hope that Kurt can say more about whether the the versions Waldek has
chosen to build an SBCL+Hunchentoot image are sufficient/compatible with
doing all the stuff jfricas does.

Thank you, Waldek, for working on the inclusion of hunchentoot.

Ralf

[Waldek Hebisch]

On Sat, Jun 10, 2023 at 11:40:58PM +0200, Ralf Hemmecke wrote:
> > > There are two problems to solve.
> > >
> > > 1) easily build from github sources (getting prerequisites on the fly)
> > > for end users that build fricas themselves
> >
> > If the user wants quicklisp I have nothing againt it as a possiblity.
> > OTOH I would like users to be able to fetch, disconnect and build.
> > Another thing is ability to recreate old versions.
>
> I have seen (but not yet tested) your hsbcl stuff.
>
> http://www.math.uni.wroc.pl/~hebisch/fricas/hsbcl.tar
>
> What I like is that you basically give explicit version numbers in the parts
> that are necessary to build hunchentoot. Yes, I support the idea of having
> concrete version numbers.
>
> Where I would be not so happy about is to put those zip (or source files)
> into the fricas repository.

Yes, I do not plan to put "zips" into repository.

> I would accept to put it into the distribution
> tarball, but that is only half of the story. A user should also be able to
> build from the git sources. So it would be good to have a script that can be
> run before "configure" in order to get those additional prerequisites.

Well, ATM hsbcl.tar tarball consists to build script + files that
quicklisp would fetch. We can put hsbcl.tar in our distribution area
(together with other tarbals) or we could put list of dependencies
in INSTALL.

I think that conceptually hunchentoot is not different than gmp:
if it is installed and we use matching lisp, then we can use it.
But it is up to user/packager to make sure that dependencies are
available. Of course, there is question of convenience: long
list of dependencies may discourage most of users. So the
tarball and build script. One could try to integrate creation
of 'hsbcl' into our build system. But after thinking a bit
about this I am not sure if this is necessary. Namely, as long
as dependencies are not part of source tree there will be extra
step or two. IMO few extra manual steps are not big hasle.
OTOH extra steps means more flexibility to users. Some
users may prefer quicklisp. Distribution packagers presumably
have their own ways and more automation on our side may
interfere with their work.

> Since jfricas is currently restricted to sbcl and FriCAS can be considered a
> something on top of quite a variety of lisps, I am actually not even sure
> whether it is right to put the hunchtoot stuff into a source distribution.
> In the binary (on top of sbcl) distribution, it would certainly make sense.

If we distribute binary containg hunchentoot, it make sense to provide
sources used to build it. One possibility is to put hunchentoot with
dependencies into "fat" distribution tarball. Probably better way is
to provide separate tarball with hunchentoot and dependencies.

> > > 2) build a distribution as a release manager
>
> > I am not sure what is current practice of Linux distributions, but
> > past standard was that all files were fetched by distribution
> > infrastructure. Network access during build was forbidden. So
> > "no network" during build was/is important for compatibility
> > with packaging FriCAS for Linux distributions.
>
> I have no problem with the no-network-paradigm. The only question is how to
> tell the user (via documentation or via a little script) how to get the
> prerequisites if "configure --enable-webspad" (i.e. jfricas) is wished. A
> user would anyway have to install jupyter himself.

Something like I wrote (where xxxx is link to repository):

wget xxxx/hsbcl-1.3.9.tar
tar -xf hsbcl-1.3.9.tar
hsbcl-1.3.9/build_hsbcl

And then appropriate --with-lisp

> > To try:
> >
> > wgethttp://www.math.uni.wroc.pl/~hebisch/fricas/hsbcl.tar
> > tar -xf hsbcl.tar
> > cd hsbcl
> > time ./build_hsbcl > rap1 2>&1
> >
> > this creates executable called 'hsbcl' which can be used to
> > build FriCAS.
>
> I've build fricas with hsbcl and no quicklisp and installed jfricas on top
> of this. The whole thing works with hsbcl and the current master.
> No additional patches from my side are required. In other words, if a user
> provides a version of SBCL that lets webspad start hunchentoot
>
> https://github.com/fricas/jfricas/blob/master/jfricas/webspad.lisp#L217
>
> then jfricas can be installed roughly as described at
> https://hemmecke.github.io/fricas/install.html#jfricas-optional
> The part with quicklisp and hunchentoot installation can be skipped.

Good to know.

--
Waldek Hebisch