Pack install from GitHub

[Julio Di Egidio]

Hello guys, a quick question:

I'm am going to use GitHub releases for a pack.

At http://www.swi-prolog.org/howto/Pack.txt we are told that creating a

GitHub release puts the file at:

    https://github.com/<owner>/<pack>/archive/<tag>.zip

But then we are told that "update checking can be enabled by setting the

download attribute of the pack to":

    https://github.com/<owner>/<pack>/release/*.zip

which is different: just wondering are the docs correct?

Thank you,

Julio

[Jan Wielemaker]

No but for a different reason. Must be (not the extra `s`):

https://github.com/<owner>/<pack>/releases/*.zip

The way it works is as follows: if the file part of the download
location contains a wildcard, it removes the file part and fetches
the HTML page (hopefully). It then parses the HTML file, looking
for links to targets that match the file pattern. It computes the
most recent version based on the version number and tries to install
that one.

Cheers --- Jan

[Julio Di Egidio]

OK, thank you.

I'll be having a problem that way, which is caused also by the fact that the

file name of a pack shall match the pack name: there could be other .zip

files than the pack files in the releases page, e.g. I mean to write my library

for both Prolog and JavaScript...  But I'll leave this problem for another day!

Thanks again for now.

Julio

[Jan Wielemaker]

On 24/08/16 10:30, Julio Di Egidio wrote:
> I'll be having a problem that way, which is caused also by the fact
> that the file name of a pack shall match the pack name: there could
> be other .zip files than the pack files in the releases page, e.g. I
> mean to write my library for both Prolog and JavaScript... But I'll
> leave this problem for another day!

That is why you have a wildcard. Just make sure that matches the Prolog
packs. And yes, this means you should make sure the archives for the
JavaScript follow a different pattern than those for the Prolog packs.
That seems easy enough.

For GitHub, the file names are just <tag>.zip, which is why it uses
simply *.zip. Maybe better V*.zip if you use VX.Y.Z tags.

Cheers --- Jan

[Julio Di Egidio]

On Wednesday, August 24, 2016 at 10:35:28 AM UTC+2, Jan Wielemaker wrote:

On 24/08/16 10:30, Julio Di Egidio wrote:
> I'll be having a problem that way, which is caused also by the fact
> that the file name of a pack shall match the pack name: there could
> be other .zip files than the pack files in the releases page, e.g. I
> mean to write my library for both Prolog and JavaScript...  But I'll
> leave this problem for another day!

That is why you have a wildcard.  Just make sure that matches the Prolog
packs.

But a disambiguation would need e.g. <name>-pl-<ver>.zip vs. <name>-js-<ver>.zip,

or similar, i.e. a pattern that "contains" the wildcard, not just the fixed pattern *.zip.

Unless I am misunderstanding you?

Julio

[Julio Di Egidio]

On Wednesday, August 24, 2016 at 10:54:56 AM UTC+2, Julio Di Egidio wrote:

On Wednesday, August 24, 2016 at 10:35:28 AM UTC+2, Jan Wielemaker wrote:

On 24/08/16 10:30, Julio Di Egidio wrote:
> I'll be having a problem that way, which is caused also by the fact
> that the file name of a pack shall match the pack name: there could
> be other .zip files than the pack files in the releases page, e.g. I
> mean to write my library for both Prolog and JavaScript...  But I'll
> leave this problem for another day!

That is why you have a wildcard.  Just make sure that matches the Prolog
packs.

But a disambiguation would need e.g. <name>-pl-<ver>.zip vs. <name>-js-<ver>.zip,

or similar, i.e. a pattern that "contains" the wildcard, not just the fixed pattern *.zip.

But again, the pack system wants a file with the same base name as the pack, hence

that catch 22 anyway...  no?

Julio

Unless I am misunderstanding you?

[Julio Di Egidio]

As said, very frustrating. E.g. here you yourself are not using "releases":

https://github.com/JanWielemaker/geoip/blob/master/pack.pl

Hence I must conclude that the wildcard is limited to the whole file name

without extension, but at least the folder path is flexible...!

I gather the way it's been implemented simply reflects those projects

where a SWI pack structure is all that there is: of course it does not scale,

but if at least the docs were more precise, one would not have to waste

so many days.

I understand SWI is a free and open and all, but really so too many features

are just concocted there based on a single in-house use case...  which BTW

doesn't save us from the total nightmare of having to presere backwards

compatibility, as on that you are inflexible, so that we are often forever stuck

with suboptimal decisions. There, I said it. Thanks anyway.

Julio

[Julio Di Egidio]

Sorry, I was  very frustrated.

I've kept looking into this.  I don't see why you make a special case for GitHub

at all.  You are doing a page scraping anyway, so the usual approach with a

free URL with just the version number as a wildcard should suffice, and just

on top of that the new automatic updates... It remains a guess: the documentation

IMO is simply inadequate, and I won't be looking into the source code.  I'll be

going the old way for the download URL, support for updates eventually should

not affect URLs.

Anyway, just my 2c.

Julio

[Julio Di Egidio]

On Tuesday, August 23, 2016 at 9:15:30 PM UTC+2, Julio Di Egidio wrote:

Hello guys, a quick question:

I'm am going to use GitHub releases for a pack.

I am not going to use GitHub for distribution, still it's definitely not my lucky day:

 ?- pack_install('http://julio.diegidio.name/Projects/Nan.Numerics.Prime/Prolog/nan_numerics_prime-1.0.zip').
ERROR: close/1: I/O error in read on stream <stream>(03C39AA0) (Invalid argument)

The web is working, and the pack installs successfully from a local file.

What is going wrong?

Julio

[Jan Wielemaker]

Dunno. It installs just fine here. Surely the error message could be
better :)
If you had used library(prolog_stack) you'd see a backtrace. I think
I'll add that
as a default feature.

Cheers --- Jan

[Jan Wielemaker]

On 08/24/2016 09:24 PM, Julio Di Egidio wrote:
>
> As said, very frustrating. E.g. here you yourself are not using "releases":
>
> https://github.com/JanWielemaker/geoip/blob/master/pack.pl

That pack file is wrong.

> Hence I must conclude that the wildcard is limited to the whole file name
> without extension, but at least the folder path is flexible...!

I explained the process:

if is_wildcard(file-name(URL)
fetch url after deleting the filename
parse HTML
find links that match the wildcard
find most recent version (a new URL)
find package name from URL
download and install

find package is basically

if is_github_url(URL)
package name is last directory segment
else
get file name part of URL
drop extension and version

> I gather the way it's been implemented simply reflects those projects
> where a SWI pack structure is all that there is: of course it does not
> scale,

I think you could say exactly the same for Python, R, node, etc. Package
managers such as Debian apt or BSD port also do more or less the same
thing, only OS specific rather than language specific.

> but if at least the docs were more precise, one would not have to waste
> so many days.

That of course is always a fair point. There is certainly a lot to wish
for wrt the package manager. The concept is good. Ideally we had a
widely adopted cross-OS cross-language package manager. I think it is
possible to build such a beast (using adapters for the OS and language),
but I'm not aware of one. As it stands, every language builds its own
and typically in the language itself to reduce external dependencies and
improve cross-OS portability. The only good news about that is that it
tests and improves the OS interaction. The bad news is that writing a
complete package manager with a sensible interface is a lot of work.
I wrote this one as a proof of concept, hoping it would evolve. Some
of that happened thanks to various users.

> I understand SWI is a free and open and all, but really so too many features

> are just concocted there based on a single in-house use case... There,

> I said it. Thanks anyway.

In part, this is certainly true. One got to start somewhere though. Much
of the stuff gets improved as it is used more widely and people both
raise issues and start extending facilities to suit their needs.

Cheers --- Jan

[Julio Di Egidio]

Thank you, I see it's already online!  :)

I'll have to fix at least one thing though, the Download URL: I have mistakenly used

https: which does not work on my site.  Let's see if I will be able to install this time...

Cheers,

Julio

P.S.  I have just seen you other post: I'll look into it.

[Julio Di Egidio]

I have created a new version of the pack, but still no luck.

Using library(prolog_stack), here is what I get:

?- pack_install('http://julio.diegidio.name/Projects/Nan.Numerics.Prime/Prolog/nan_numerics_prime-1.0.1.zip').
ERROR: I/O error in read on stream <stream>(0431C558) (Invalid argument)
ERROR: In:
ERROR:   [12] setup_call_catcher_cleanup(prolog_pack: ...,prolog_pack: ...,exit,prolog_pack: ...)
ERROR:   [10] prolog_pack:pack_install_from_url(http,'http://julio.diegidio.name/Projects/Nan.Numerics.Prime/Prolog/nan_numerics_prime-1.0.1.zip','c:/program files (x86)/swipl/pack',nan_numerics_prime,[...|...]) at c:/program files (x86)/swipl/library/prolog_pack.pl:796

The .zip file is downloaded successfully to folder pack/Downloads/ (it's there and I can

open it fine), and the folder for the pack is created, too, in folder pack/, but no file is in it.

OTOH, as yesterday, the pack installs just fine if I use a local file.

I won't mind if you simply install the new pack for me, but if I can be of any assistance

narrowing this down please just let me know what I can do or try.

As a side issue, not critical though: I have noticed something is weird with the

online documentation system, namely, if I go to the pack on SWI web site, there

is a list of contained files and opening one of the prolog files brings a doc 

page.  Few things: documented predicates are not in the original order (this does

not happen via a doc_server locally); the show private predicates button does

nothing; and the view source button gives an HTTP 500 server error most times

(not always though).

Julio