Bug: source_sink does not exist

[Volker Wysk]

Hi

I've found another bug:

$ mkdir -p /tmp/foo/bar
$ cd /tmp/foo
$ /usr/local/bin/swipl
Welcome to SWI-Prolog (threaded, 64 bits, version 7.7.12-89-g347a3cc)
SWI-Prolog comes with ABSOLUTELY NO WARRANTY. This is free software.
Please run ?- license. for legal details.

For online help and background, visit http://www.swi-prolog.org
For built-in help, use ?- help(Topic). or ?- apropos(Word).

?- working_directory(_, '/tmp/foo'), absolute_file_name('bar', Abs, [relative_to('/tmp/foo')]).
ERROR: source_sink `bar' does not exist
ERROR: In:
ERROR: [12] throw(error(existence_error(source_sink,bar),_2456))
ERROR: [7] <user>
ERROR:
ERROR: Note: some frames are missing due to last-call optimization.
ERROR: Re-run your program in debug mode (:- debug.) to get more detail.

$ /usr/local/bin/swipl --version
SWI-Prolog version 7.7.12 for x86_64-linux


Bye

[Paulo Moura]

Not a bug. By default, the SWI-Prolog (and also YAP) absolute_file_name/3 predicate only expands files. To also expand directories, you need to add the file_type(directory) option to the third argument.

> --
> You received this message because you are subscribed to the Google Groups "SWI-Prolog" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to swi-prolog+...@googlegroups.com.
> Visit this group at https://groups.google.com/group/swi-prolog.
> For more options, visit https://groups.google.com/d/optout.

[Jan Wielemaker]

On 27/04/18 16:24, Paulo Moura wrote:
> Not a bug. By default, the SWI-Prolog (and also YAP) absolute_file_name/3 predicate only expands files. To also expand directories, you need to add the file_type(directory) option to the third argument.

Yip. Not _also_ though. If you give file_type(directory) is only finds
directories. That is because you use absolute_file_name/3 if you know
what you are looking for and how you want to use it but you don't
know where to find it.

Cheers --- Jan

[Paulo Moura]

Hi Jan,

> On 27 Apr 2018, at 15:27, Jan Wielemaker <j...@swi-prolog.org> wrote:
>
> On 27/04/18 16:24, Paulo Moura wrote:
>> Not a bug. By default, the SWI-Prolog (and also YAP) absolute_file_name/3 predicate only expands files. To also expand directories, you need to add the file_type(directory) option to the third argument.
>
> Yip. Not _also_ though.

Hum? YAP 6.3 or later behaves the same. In fact, I use the sam exact definition around absolute_file_name/3 to expand paths internally in Logtalk for both systems:

'$lgt_expand_path'(Path, ExpandedPath) :-
working_directory(Current, Current),
( absolute_file_name(Path, [expand(true), relative_to(Current), file_errors(fail)], ExpandedPath) ->
true
; absolute_file_name(Path, [expand(true), relative_to(Current), file_type(directory), file_errors(fail)], ExpandedPath)
).

Volker, maybe you can use something similar?

Cheers,
Paulo

[Jan Wielemaker]

On 27/04/18 16:32, Paulo Moura wrote:
> Hi Jan,
>
>> On 27 Apr 2018, at 15:27, Jan Wielemaker <j...@swi-prolog.org> wrote:
>>
>> On 27/04/18 16:24, Paulo Moura wrote:
>>> Not a bug. By default, the SWI-Prolog (and also YAP) absolute_file_name/3 predicate only expands files. To also expand directories, you need to add the file_type(directory) option to the third argument.
>>
>> Yip. Not _also_ though.
>
> Hum? YAP 6.3 or later behaves the same. In fact, I use the sam exact definition around absolute_file_name/3 to expand paths internally in Logtalk for both systems:

Maybe Vitor also came to the conclusion it is not so nice to get a file
if you ask for a directory, in particular if the search path both
contains a file and a directory with the target name.

>
> '$lgt_expand_path'(Path, ExpandedPath) :-
> working_directory(Current, Current),
> ( absolute_file_name(Path, [expand(true), relative_to(Current), file_errors(fail)], ExpandedPath) ->
> true
> ; absolute_file_name(Path, [expand(true), relative_to(Current), file_type(directory), file_errors(fail)], ExpandedPath)
> ).

Dunno about YAP, but from its Quintus origin absolute_file_name/3
already resolves files relative to the current working directory.
The relative_to option was added by me as some point to avoid
the need to change working directory while compiling files. Changing
working directory is a particularly bad idea in threaded applications
as this notion is shared. You change WD at most once at startup.
You do want searches for files loaded from some file to be relative
to the loading file though.

In my experience it is extremely rare to search for either a file or
a directory while using absolute_file_name/3. That suggests the
abstraction is not good or I'm an a-typical user. If that is the
case and this dual search is common we should have some option.
Searching the file system twice should not be needed.

Cheers --- Jan

[Paulo Moura]

> On 27 Apr 2018, at 15:44, Jan Wielemaker <j...@swi-prolog.org> wrote:
>
> On 27/04/18 16:32, Paulo Moura wrote:
>> Hi Jan,
>>> On 27 Apr 2018, at 15:27, Jan Wielemaker <j...@swi-prolog.org> wrote:
>>>
>>> On 27/04/18 16:24, Paulo Moura wrote:
>>>> Not a bug. By default, the SWI-Prolog (and also YAP) absolute_file_name/3 predicate only expands files. To also expand directories, you need to add the file_type(directory) option to the third argument.
>>>
>>> Yip. Not _also_ though.
>> Hum? YAP 6.3 or later behaves the same. In fact, I use the sam exact definition around absolute_file_name/3 to expand paths internally in Logtalk for both systems:
>
> Maybe Vitor also came to the conclusion it is not so nice to get a file if you ask for a directory, in particular if the search path both contains a file and a directory with the target name.

No idea of the exact date where the YAP behavior changed but the change is at least several years old.

>> '$lgt_expand_path'(Path, ExpandedPath) :-
>> working_directory(Current, Current),
>> ( absolute_file_name(Path, [expand(true), relative_to(Current), file_errors(fail)], ExpandedPath) ->
>> true
>> ; absolute_file_name(Path, [expand(true), relative_to(Current), file_type(directory), file_errors(fail)], ExpandedPath)
>> ).
>
> Dunno about YAP, but from its Quintus origin absolute_file_name/3 already resolves files relative to the current working directory.

Indeed, I can simplify the above predicate for SWI-Prolog assuming that that default is true since at least SWI-Prolog 6.6.0? Can you confirm? Thanks.

> The relative_to option was added by me as some point to avoid
> the need to change working directory while compiling files. Changing
> working directory is a particularly bad idea in threaded applications
> as this notion is shared. You change WD at most once at startup.
> You do want searches for files loaded from some file to be relative
> to the loading file though.
>
> In my experience it is extremely rare to search for either a file or
> a directory while using absolute_file_name/3. That suggests the
> abstraction is not good or I'm an a-typical user.

Note that absolute_file_name/2-3 are far from standard, which in the particular case of Logtalk (supporting 12 Prolog systems) resulted in that particular *internal* predicate whose definition is only shared by 2 of the 12 systems.

> If that is the
> case and this dual search is common we should have some option.
> Searching the file system twice should not be needed.

That would be nice in my particular case. But not sure how common the usage of such an option would be.

Cheers,
Paulo

[Jan Wielemaker]

On 27/04/18 17:05, Paulo Moura wrote:
> Indeed, I can simplify the above predicate for SWI-Prolog assuming
> that that default is true since at least SWI-Prolog 6.6.0? Can you
> confirm? Thanks.

I don't think it has ever been different. Only relative_to was added
later. Probably when threads were added. Before that, the system
changed working directory for loading files.

>> The relative_to option was added by me as some point to avoid the
>> need to change working directory while compiling files. Changing
>> working directory is a particularly bad idea in threaded
>> applications as this notion is shared. You change WD at most once
>> at startup. You do want searches for files loaded from some file to
>> be relative to the loading file though.
>>
>> In my experience it is extremely rare to search for either a file
>> or a directory while using absolute_file_name/3. That suggests
>> the abstraction is not good or I'm an a-typical user.
>
> Note that absolute_file_name/2-3 are far from standard, which in the
> particular case of Logtalk (supporting 12 Prolog systems) resulted in

> that particular*internal* predicate whose definition is only shared

> by 2 of the 12 systems.

At least Quintus, SICStus, YAP and SWI-Prolog have it :) More should
as it is one of these really useful features invented at Quintus.

>> If that is the case and this dual search is common we should have
>> some option. Searching the file system twice should not be needed.
>
> That would be nice in my particular case. But not sure how common the
> usage of such an option would be.

Depends. If your the use of your abstraction is or should have been
followed by a test for file/directory, I think you should change the
abstraction to ask for a file or directory. If you have a common use
case that you want to find something by its name and you do not care
whether you get a file or directory we should reconsider.

Cheers --- Jan

[Paulo Moura]

> On 27 Apr 2018, at 16:14, Jan Wielemaker <j...@swi-prolog.org> wrote:
>
> On 27/04/18 17:05, Paulo Moura wrote:
>> Indeed, I can simplify the above predicate for SWI-Prolog assuming
>> that that default is true since at least SWI-Prolog 6.6.0? Can you
>> confirm? Thanks.
>
> I don't think it has ever been different. Only relative_to was added
> later. Probably when threads were added. Before that, the system
> changed working directory for loading files.

Just found a copy of the 6.6.6 version manual. Indeed the same behavior is described there. And no, I don't keep a copy of that specific version manual for some satanic reason :-D

>>> The relative_to option was added by me as some point to avoid the
>>> need to change working directory while compiling files. Changing
>>> working directory is a particularly bad idea in threaded
>>> applications as this notion is shared. You change WD at most once
>>> at startup. You do want searches for files loaded from some file to
>>> be relative to the loading file though.
>>>
>>> In my experience it is extremely rare to search for either a file
>>> or a directory while using absolute_file_name/3. That suggests
>>> the abstraction is not good or I'm an a-typical user.
>>
>> Note that absolute_file_name/2-3 are far from standard, which in the
>> particular case of Logtalk (supporting 12 Prolog systems) resulted in
>> that particular*internal* predicate whose definition is only shared
>> by 2 of the 12 systems.
>
> At least Quintus, SICStus, YAP and SWI-Prolog have it :) More should
> as it is one of these really useful features invented at Quintus.

Having the predicate doesn't mean, unfortunately, that the predicate behaves the same. For example, using Volker's test case with the latest SICStus Prolog version:

$ mkdir -p /tmp/foo/bar
$ cd /tmp/foo

$ sicstus
SICStus 4.4.1 (x86_64-darwin-17.2.0): Sat Mar 17 12:57:47 CET 2018
Licensed to Paulo Moura
| ?- use_module(library(file_systems)).
...
yes
| ?- current_directory(_, '/tmp/foo'), absolute_file_name('bar', Abs, [relative_to('/tmp/foo')]).
Abs = '/tmp/foo/bar' ?
yes

Both system also support a file_errors/1 option. But they have different semantics. Thus, for SICStus Prolog, I can simply use:

'$lgt_expand_path'(Path, ExpandedPath) :-
absolute_file_name(Path, ExpandedPath).

YAP seems to have changed behavior for being close to SICStus Prolog to be close to SWI-Prolog when moving from 6.2 to 6.3 as I use:

:- if((current_prolog_flag(version_data, yap(Major,Minor,_,_)), (Major,Minor) @< (6,3))).

'$lgt_expand_path'(Path, ExpandedPath) :-
working_directory(Current, Current),

absolute_file_name(Path, [access(none), file_type(txt), relative_to(Current)], ExpandedPath).
:- else.
'$lgt_expand_path'(Path, ExpandedPath) :-
( absolute_file_name(Path, [expand(true), file_errors(fail)], ExpandedPath) ->
true
; absolute_file_name(Path, [expand(true), file_type(directory), file_errors(fail)], ExpandedPath)
).
:- endif.

Portability is a freaking nightmare but we already know that.

>>> If that is the case and this dual search is common we should have
>>> some option. Searching the file system twice should not be needed.
>>
>> That would be nice in my particular case. But not sure how common the
>> usage of such an option would be.
>
> Depends. If your the use of your abstraction is or should have been
> followed by a test for file/directory, I think you should change the
> abstraction to ask for a file or directory. If you have a common use
> case that you want to find something by its name and you do not care
> whether you get a file or directory we should reconsider.

From a quick verification (of the Logtalk adapter files for the 12 supported systems), most systems expand a path irrespective of if we are looking for a (regular) file or a directory.

Cheers,
Paulo

[Volker Wysk]

Am Freitag, 27. April 2018, 15:32:38 CEST schrieb Paulo Moura:
> >> Not a bug. By default, the SWI-Prolog (and also YAP) absolute_file_name/3 predicate only expands files. To also expand directories, you need to add the file_type(directory) option to the third argument.
> >
> > Yip. Not _also_ though.
>
> Hum? YAP 6.3 or later behaves the same. In fact, I use the sam exact definition around absolute_file_name/3 to expand paths internally in Logtalk for both systems:
>
> '$lgt_expand_path'(Path, ExpandedPath) :-
> working_directory(Current, Current),
> ( absolute_file_name(Path, [expand(true), relative_to(Current), file_errors(fail)], ExpandedPath) ->
> true
> ; absolute_file_name(Path, [expand(true), relative_to(Current), file_type(directory), file_errors(fail)], ExpandedPath)
> ).
>
> Volker, maybe you can use something similar?

What is "YAP"? And Logtalk..?

It works for now. But I'd rather use "realpath", if it was available.

Volker

[Volker Wysk]

Am Freitag, 27. April 2018, 15:24:06 CEST schrieb Paulo Moura:
> Not a bug. By default, the SWI-Prolog (and also YAP) absolute_file_name/3 predicate only expands files. To also expand directories, you need to add the file_type(directory) option to the third argument.

Okay, no bug. But that error message should be improved:

ERROR: source_sink `bar' does not exist

What is a "source_sink"? What does it mean when it doesn't exist?


Bye
Volker

[Jan Wielemaker]

source_sink is the official ISO Prolog term for something you can open
as a stream. And indeed, there is no file bar in the dir /tmp/foo.
Not sure how to improve much as the error term is dictated by the
standard (well, almost as absolute_file_name/3 is not in the standard).

Cheers --- Jan

[Volker Wysk]

Am Samstag, 28. April 2018, 14:45:21 CEST schrieb Jan Wielemaker:
> Not sure how to improve much as the error term is dictated by the
> standard (well, almost as absolute_file_name/3 is not in the standard).

The error message is dictated? (the "error term"?) Really ...

Perhaps, one could add something to the message. Such as an URL to the relevant part of the standard..? Or a message like "See the prolog standard for more information".

Bye,
Volker

[Jan Wielemaker]

On 28/04/18 16:06, Volker Wysk wrote:
> Am Samstag, 28. April 2018, 14:45:21 CEST schrieb Jan Wielemaker:
>> Not sure how to improve much as the error term is dictated by the
>> standard (well, almost as absolute_file_name/3 is not in the
>> standard).
>
> The error message is dictated? (the "error term"?) Really ...

The standard says that BIP (Built in predicates) produce exceptions of
the form error(Formal, ImplementationDependent), where Formal is
dictated by the standard and ImplementationDependent is left to the
implementation. Formal are a number of terms, one of which is
existence_error(TypeOfObject, Object). It defines a number of
TypeOfObject values, one of which is `source_sink` which (I my
interpretation) is an abstraction of "file".

Of course the standard can only describe the errors for defined
BIPs. I (and most other Prologs AFAIK) try to reuse the standard
error terms as much as possible for other BIPS. That leaves little
choice but raising

error(existence_error(source_sink, bar),
ImplementationDependent)

ImplementationDependent varies wildly between systems. In SWI-Prolog the
code raising the error typically leaves it unbound or binds it to
context(_, Message). The library(prolog_stack) decorates the exception
with a stack trace if it is uncaught and unifies with the shape
context(Stack, _). Finally, print_message/2 turns all this into a
message for the console.

So yes, in theory the system could add additional information in the
Message part of the context term. This is used sparsely. Possibly too
sparsely. A big obstacle is that this is really a lot of work and
requires a lot of thought to get something consistent. If you want to be
consistent you probably have to catch and rethrow errors in higher BIPs
to make the message fit the context. That is a lot of code that makes
the sources less readable and the system bigger and slower.

This is for programmers, so in how many ways can you interpret the fact
that absolute_file_name(bar, ...) tells you that the `source_sink` `bar`
doesn't exist?

I surely agree that interpreting error messages is not really easy and
typically highly system specific. What about "Warning: your head is
detached" by GIT :) It is also true that systems are getting better
in error messages and that has a lot of value to novice users.

My intuition would tell me that the way to go is to write a library that
translates the entire error term (formal, stack and message) to
something nice and crisp for novices. That is quite a challenge though.
In this case the rule set could reason that source_sink in the context
of absolute_file_name/3 means "file" (or "directory" if the
file_type(directory) was given) and the "file" "bar" doesn't exist while
"bar" exists but is a directory rather than a file. So, we get:

ERROR: The regular file "bar" doesn't exist. "bar" exists as a
directory.
ERROR: Use the option file_type(directory) to match directories.

It would be great to have something like that!

Cheers --- Jan

[Volker Wysk]

Am Samstag, 28. April 2018, 18:45:16 CEST schrieb Jan Wielemaker:
>
> So yes, in theory the system could add additional information in the
> Message part of the context term. This is used sparsely. Possibly too
> sparsely. A big obstacle is that this is really a lot of work and
> requires a lot of thought to get something consistent. If you want to be
> consistent you probably have to catch and rethrow errors in higher BIPs
> to make the message fit the context. That is a lot of code that makes
> the sources less readable and the system bigger and slower.

My idea was to add a reference (best would be an URL) to the Prolog standard, to where those error messages are defined. Only one piece of text for all kinds of errors.

I've searched for the ISO prolog standard on the web, but there appears to be no copy online. Instead, they're *selling* the standard at iso.org, and it's quite expensive.

The closest I could find is:

ISO Prolog: A Summary of the Draft Proposed Standard
http://fsl.cs.illinois.edu/images/9/9c/PrologStandard.pdf

ISO conformant version of the Prolog language
http://www.deransart.fr/prolog/docs.html

With such a reference, you could figure out, what that error message means, even if you're a beginner.

> This is for programmers, so in how many ways can you interpret the fact
> that absolute_file_name(bar, ...) tells you that the `source_sink` `bar`
> doesn't exist?

I thought of "source code", when I read "source sink"... What is a "sink"-?

> I surely agree that interpreting error messages is not really easy and
> typically highly system specific. What about "Warning: your head is
> detached" by GIT :) It is also true that systems are getting better
> in error messages and that has a lot of value to novice users.
>
> My intuition would tell me that the way to go is to write a library that
> translates the entire error term (formal, stack and message) to
> something nice and crisp for novices. That is quite a challenge though.
> In this case the rule set could reason that source_sink in the context
> of absolute_file_name/3 means "file" (or "directory" if the
> file_type(directory) was given) and the "file" "bar" doesn't exist while
> "bar" exists but is a directory rather than a file. So, we get:
>
> ERROR: The regular file "bar" doesn't exist. "bar" exists as a
> directory.
> ERROR: Use the option file_type(directory) to match directories.
>
> It would be great to have something like that!

Yes, that would be ideal. But a small hint on more information on how to interpret the error message, would also be useful.

Cheers
Volker

[Volker Wysk]

Am Sonntag, 29. April 2018, 11:25:37 CEST schrieb Volker Wysk:
> My idea was to add a reference (best would be an URL) to the Prolog standard, to where those error messages are defined. Only one piece of text for all kinds of errors.

I mean, "refering to the Prolog standard", not "adding it to the Prolog Standard" :-)

Volker