RE: gather-info, msnoticefile, and dumpdatabase
1 view
Skip to first unread message
Larry Masinter
Jun 9, 2021, 4:54:43 PM6/9/21
to Ron Kaplan, Interlisp core
I had GATHER-INFO on the file internal/library/MEDLEY-UTILS
But I thought it wasn't stable.... the "info" it gathers is incomplete and I hadn't figured out what anomalies really need to be reported / repaired.
Now I'm wondering if we should make LISPUSERS a different repo with a different commit policy. Or if there were some kind of sync service we could use for sharing personal init files....
The thing that struck me but I wasn't sure how to repair was "DUPFNS" -- functions with different definitions on different files. Other things to look for is missing GLOBALVARS declarations.
Why MSNOTICEFILE doesn't work is a puzzle.
(for X in SYSFILES do (MASTERSCOPE `(ANALYZE ON ,X))
--
https://LarryMasinter.net https://interlisp.org
> -----Original Message-----
> From: Ron Kaplan <ron.k...@post.harvard.edu>
> Sent: Wednesday, June 9, 2021 7:33 AM
> To: Larry Masinter <l...@acm.org>
> Subject: gather-info, msnoticefile, and dumpdatabase
>
> Perhaps that’s something you should make as a separate little lispusers
> package? (And maybe generalized with an optional argument to specify a
> particular set of files of interest?)
>
> On the Masterscope, it does the MSNOTICEFILE but doesn’t analyze. Should
> that be added as a separate tasks (i.e. 1 to 5 instead of 1 to 4), or done
> automatically as task 4?
>
> Finally, after noticing all the files, explicitly analyzing them, and dumping the
> database (DUMPDATABASE NIL ‘full.database), I load the database into a
> fresh system. I then ask . WHO CALLS \OUTCHAR (lots of results), followed
> by . WHO CONTAINS THOSE (nothing).
>
> So it appears that DUMPDATABASE does not dump whatever information is
> instantiated when a file is noticed.
>
> Should DUMPDATABASE be extended so that it automatically saves that
> additional information? (Maybe just FILEFNSLST for each of the noticed files?
> Anything else?)
>
> Alternatively, since my extension makes up its own PRETTYDEF command on
> the fly, I could have it dump the database (E (DUMPDATABASE)) and then
> dump the notice information separately (a VARS command?) so that would
> also be restored when the file is LOADed.
>
>
>
But I thought it wasn't stable.... the "info" it gathers is incomplete and I hadn't figured out what anomalies really need to be reported / repaired.
Now I'm wondering if we should make LISPUSERS a different repo with a different commit policy. Or if there were some kind of sync service we could use for sharing personal init files....
The thing that struck me but I wasn't sure how to repair was "DUPFNS" -- functions with different definitions on different files. Other things to look for is missing GLOBALVARS declarations.
Why MSNOTICEFILE doesn't work is a puzzle.
(for X in SYSFILES do (MASTERSCOPE `(ANALYZE ON ,X))
--
https://LarryMasinter.net https://interlisp.org
> -----Original Message-----
> From: Ron Kaplan <ron.k...@post.harvard.edu>
> Sent: Wednesday, June 9, 2021 7:33 AM
> To: Larry Masinter <l...@acm.org>
> Subject: gather-info, msnoticefile, and dumpdatabase
>
> Perhaps that’s something you should make as a separate little lispusers
> package? (And maybe generalized with an optional argument to specify a
> particular set of files of interest?)
>
> On the Masterscope, it does the MSNOTICEFILE but doesn’t analyze. Should
> that be added as a separate tasks (i.e. 1 to 5 instead of 1 to 4), or done
> automatically as task 4?
>
> Finally, after noticing all the files, explicitly analyzing them, and dumping the
> database (DUMPDATABASE NIL ‘full.database), I load the database into a
> fresh system. I then ask . WHO CALLS \OUTCHAR (lots of results), followed
> by . WHO CONTAINS THOSE (nothing).
>
> So it appears that DUMPDATABASE does not dump whatever information is
> instantiated when a file is noticed.
>
> Should DUMPDATABASE be extended so that it automatically saves that
> additional information? (Maybe just FILEFNSLST for each of the noticed files?
> Anything else?)
>
> Alternatively, since my extension makes up its own PRETTYDEF command on
> the fly, I could have it dump the database (E (DUMPDATABASE)) and then
> dump the notice information separately (a VARS command?) so that would
> also be restored when the file is LOADed.
>
>
>
Michele Denber
Jun 9, 2021, 5:32:19 PM6/9/21
to lisp...@googlegroups.com
I have some 3-dimensioonal Common Lisp arrays I want to save to a file
and later read back in. Does Lisp have a way to do this or do I have to
roll my own? I tried (PRINT MYARRAY FILE) and all that does is save the
name of the pointer to the array, not the contents. Any advice would be
appreciated.
- Michele
and later read back in. Does Lisp have a way to do this or do I have to
roll my own? I tried (PRINT MYARRAY FILE) and all that does is save the
name of the pointer to the array, not the contents. Any advice would be
appreciated.
- Michele
Ron Kaplan
Jun 9, 2021, 5:45:30 PM6/9/21
to Larry Masinter, Interlisp core
I think Lispusers always had a different service-level stipulation: may be useful, but use at your own risk.
I looked at MSNOTICEFILE. It seems that all it does is to find the file, and then do a LOADFROM or LOADVARS to make sure anything that might be needed for analysis has been loaded. But it doesn’t do any analysis.
It does push the name of the file and the name of its coms (not the coms) on a list, so it has a record of what it is loaded. But according to our masterscope database, nobody else looks at that variable.
But that’s all.
I have been poking around. There is a table on MSDATABASELST for CONTAINS, but it is empty after the full.database is loaded.
I don’t know if it had any thing when the database was built. Maybe the table was populated but not dumped. I’ll check that. It would probably be easier to fix the dumping then to figure out how to construct the table.
There is a function ONFILE which seems like it does the contains checking. It has a comment that it is worried that an MSHASHFILE might not have that table, but otherwise it does seem to test that (empty) table. And at that point it seems to be driven off the filecoms of files that happen to be incore, which would require doing a MSNOTICEFILE again on all the files in the current environment, which is what we want to avoid.
Another brute-force solution would be to put in a call to WHEREIS(.. T..), as a last-ditch answer to WHO CONTAINS.
I looked at MSNOTICEFILE. It seems that all it does is to find the file, and then do a LOADFROM or LOADVARS to make sure anything that might be needed for analysis has been loaded. But it doesn’t do any analysis.
It does push the name of the file and the name of its coms (not the coms) on a list, so it has a record of what it is loaded. But according to our masterscope database, nobody else looks at that variable.
But that’s all.
I have been poking around. There is a table on MSDATABASELST for CONTAINS, but it is empty after the full.database is loaded.
I don’t know if it had any thing when the database was built. Maybe the table was populated but not dumped. I’ll check that. It would probably be easier to fix the dumping then to figure out how to construct the table.
There is a function ONFILE which seems like it does the contains checking. It has a comment that it is worried that an MSHASHFILE might not have that table, but otherwise it does seem to test that (empty) table. And at that point it seems to be driven off the filecoms of files that happen to be incore, which would require doing a MSNOTICEFILE again on all the files in the current environment, which is what we want to avoid.
Another brute-force solution would be to put in a call to WHEREIS(.. T..), as a last-ditch answer to WHO CONTAINS.
Ron Kaplan
Jun 9, 2021, 5:46:10 PM6/9/21
to Michele Denber, lisp...@googlegroups.com
Try HPRINT. And perhaps the filepkg command HORRIBLEVARS
> On Jun 9, 2021, at 2:32 PM, Michele Denber <mde...@gmx.com> wrote:
>
> I have some 3-dimensioonal Common Lisp arrays I want to save to a file and later read back in. Does Lisp have a way to do this or do I have to roll my own? I tried (PRINT MYARRAY FILE) and all that does is save the name of the pointer to the array, not the contents. Any advice would be appreciated.
>
> - Michele
>
>
> --
> You received this message because you are subscribed to the Google Groups "Medley Interlisp core" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to lispcore+u...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/lispcore/931a78a8-a70a-a5f7-cbff-e3f2b8f80ca1%40gmx.com.
> On Jun 9, 2021, at 2:32 PM, Michele Denber <mde...@gmx.com> wrote:
>
> I have some 3-dimensioonal Common Lisp arrays I want to save to a file and later read back in. Does Lisp have a way to do this or do I have to roll my own? I tried (PRINT MYARRAY FILE) and all that does is save the name of the pointer to the array, not the contents. Any advice would be appreciated.
>
> - Michele
>
>
> You received this message because you are subscribed to the Google Groups "Medley Interlisp core" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to lispcore+u...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/lispcore/931a78a8-a70a-a5f7-cbff-e3f2b8f80ca1%40gmx.com.
John Cowan
Jun 9, 2021, 5:52:56 PM6/9/21
to Michele Denber, Interlisp core
I think you want CL:PRINT, as CL and IL arrays are different and incompatible. That should give you #3A(...), where ... is a nested list of depth 3. Of course you will have to use CL:READ to get it back.
Michele Denber
Jun 9, 2021, 8:46:07 PM6/9/21
to John Cowan, Interlisp core
On 06-09-2021 5:52 PM, John Cowan
wrote:
I think you want CL:PRINT, as CL and IL arrays are different and incompatible. That should give you #3A(...), where ... is a nested list of depth 3. Of course you will have to use CL:READ to get it back.
Thanks for the suggestions. CL:PRINT gave the same result as just
plain PRINT. HPRINT gave an error "Cannot print this item
|<ARRAY T (6 16 4) # 171,66702>. CL:HPRINT is an error. I
read about HORRIBLEVARS and didn't get it. It seems to be something
to do with MAKEFILE. I want to write a file from inside a running
function. I'm looking for something like the Matlab command
writematrix (A). I did a web search and couldn't find an answer.
The closest I could come was this:
https://stackoverflow.com/questions/9106289/reading-an-array-from-a-text-file-in-common-lisp/9107634
which basically suggests writing each element one at a time in a loop. Ugh.
- Michele
https://stackoverflow.com/questions/9106289/reading-an-array-from-a-text-file-in-common-lisp/9107634
which basically suggests writing each element one at a time in a loop. Ugh.
- Michele
John Cowan
Jun 9, 2021, 10:11:28 PM6/9/21
to Michele Denber, Interlisp core
CL:PRINT should output something like this:
#3A(((1 2) (10 20)) ((3 4) (30 40))))
This is a CL:ARRAY, a 2 x 2 x 2 cube. An IL:ARRAY is one-dimensional and has no external representation.
Ron Kaplan
Jun 9, 2021, 11:01:11 PM6/9/21
to Larry Masinter, Interlisp core
I determined that the CONTAINS table is not constructed during the analysis, whether or not it would eventually be dumped.
On the DUPFNS: there is another issue that ought to be addressed, that we see clearly with the Masterscope analysis. That is that there are many duplicate field names in various system and package record declarations, and the fetch/replace of those fields is not qualified by the name of the record. Incrementally, those qualifications should be put in.
And then a separate problem I am now seeing in the Masterscop analysis: When it comes to the file RECORD, inside the analysis Masterscope seems to generate MSNOTICEFILE calls not for the names of files, but for various random system records. And of course it then breaks because those files don’t exist.
On the DUPFNS: there is another issue that ought to be addressed, that we see clearly with the Masterscope analysis. That is that there are many duplicate field names in various system and package record declarations, and the fetch/replace of those fields is not qualified by the name of the record. Incrementally, those qualifications should be put in.
And then a separate problem I am now seeing in the Masterscop analysis: When it comes to the file RECORD, inside the analysis Masterscope seems to generate MSNOTICEFILE calls not for the names of files, but for various random system records. And of course it then breaks because those files don’t exist.
Larry Masinter
Jun 9, 2021, 11:42:33 PM6/9/21
to Ron Kaplan, Interlisp core
There's a problem with relying too much on Masterscope responses about "negatives".
Sometimes variables are evaluated using GETTOPVAL. Sometimes functions
are called using APPLY. You might confirm that the CONTAINS table isn't used
by
EDITMODE(TELETYPE)
(SETQ MAXLEVEL (SETQ MAXLOOP 30000))
ED(ALLCONTENT VARS)
F MASTERSCOPE-VAR
With whatever variable you want to make sure isn't referenced.
I had a fix to the RECORD analyze file problem. Now, I don't
remember, but I think it was just loading SYSEDIT or making sure MACRO-FN
was on MSMACROPROPS.
There was a problem with GETTEMPLATE creating a lot of "Masterscope templates"
that were bogus or wrong. That might have been the MSMACROPROPS too.
Or it might have been some free variable problem running GATHER-INFO
interpreted instead of compiled.
I vaguely remember the record package giving priority to records on the
same file, but it's been a long time.
> 2D35B5FCBCDF%40post.harvard.edu.
Sometimes variables are evaluated using GETTOPVAL. Sometimes functions
are called using APPLY. You might confirm that the CONTAINS table isn't used
by
EDITMODE(TELETYPE)
(SETQ MAXLEVEL (SETQ MAXLOOP 30000))
ED(ALLCONTENT VARS)
F MASTERSCOPE-VAR
With whatever variable you want to make sure isn't referenced.
I had a fix to the RECORD analyze file problem. Now, I don't
remember, but I think it was just loading SYSEDIT or making sure MACRO-FN
was on MSMACROPROPS.
There was a problem with GETTEMPLATE creating a lot of "Masterscope templates"
that were bogus or wrong. That might have been the MSMACROPROPS too.
Or it might have been some free variable problem running GATHER-INFO
interpreted instead of compiled.
I vaguely remember the record package giving priority to records on the
same file, but it's been a long time.
> --
> You received this message because you are subscribed to the Google Groups
> "Medley Interlisp core" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to lispcore+u...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/lispcore/713C6E45-7620-44CF-8902-
> You received this message because you are subscribed to the Google Groups
> "Medley Interlisp core" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to lispcore+u...@googlegroups.com.
> To view this discussion on the web visit
> 2D35B5FCBCDF%40post.harvard.edu.
Ron Kaplan
Jun 10, 2021, 1:04:31 AM6/10/21
to Larry Masinter, Interlisp core
In the case of MSFILELST, it doesn’t show up anywhere else in ALLCONTENT.
This is the first time I’ve actually used the TTY editor since I was young. Don’t remember how to back up so that I can see the context. What worked better for me was to map ALLCONTENTS with a trivial little search function that I have, call EDITE (SEDIT) on the result, and then poke around.
I tried loading SYSEDIT first. Analyzing the file RECORD still started to think of records as files it needs to analyze.
That didn’t happen the first time I ran this, I don’t know what’s different.
This is the first time I’ve actually used the TTY editor since I was young. Don’t remember how to back up so that I can see the context. What worked better for me was to map ALLCONTENTS with a trivial little search function that I have, call EDITE (SEDIT) on the result, and then poke around.
I tried loading SYSEDIT first. Analyzing the file RECORD still started to think of records as files it needs to analyze.
That didn’t happen the first time I ran this, I don’t know what’s different.
Michele Denber
Jun 10, 2021, 12:03:10 PM6/10/21
to John Cowan, Interlisp core
On 06-09-2021 10:11 PM, John Cowan
wrote:
CL:PRINT should output something like this:
#3A(((1 2) (10 20)) ((3 4) (30 40))))
This is a CL:ARRAY, a 2 x 2 x 2 cube. An IL:ARRAY is one-dimensional and has no external representation.
I must be doing something wrong then because trying it with just a
plain CL:CPRINT without even bothering with the file part:
(SETQ BDISC (CL:MAKE-ARRAY '(6 14 4)))
#<ARRAY T (6 16 4) 0 171,88856>
(CL:SETF (CL:AREF BDISC 5 3 3) 1)
1
(CL:PRINT BDISC)
#<ARRAY T (6 16 4) 0 171,88856>
#<ARRAY T (6 16 4) 0 171,88856>
Is this right?
- Michele
(SETQ BDISC (CL:MAKE-ARRAY '(6 14 4)))
#<ARRAY T (6 16 4) 0 171,88856>
(CL:SETF (CL:AREF BDISC 5 3 3) 1)
1
(CL:PRINT BDISC)
#<ARRAY T (6 16 4) 0 171,88856>
#<ARRAY T (6 16 4) 0 171,88856>
Is this right?
- Michele
Michele Denber
Jun 10, 2021, 1:00:23 PM6/10/21
to lisp...@googlegroups.com
OK, never mind. It turns out that to print out the contents of a CL
array, you need the CL variable *print-array* set to T. In my
xfull35.sysout, it was initially NIL. It's working now :-)
- Michele
array, you need the CL variable *print-array* set to T. In my
xfull35.sysout, it was initially NIL. It's working now :-)
- Michele
Larry Masinter
Jul 18, 2021, 2:18:04 PM7/18/21
to Ron Kaplan, Interlisp core
Could someone turn this into Issues (at least for things still open?)
Thanks
Thanks
> From: lisp...@googlegroups.com <lisp...@googlegroups.com> On Behalf
> Of Ron Kaplan
> Sent: Wednesday, June 9, 2021 8:01 PM
> To: Larry Masinter <L...@acm.org>
> Cc: Interlisp core <lisp...@googlegroups.com>
> Subject: Re: gather-info, msnoticefile, and dumpdatabase
>
> Of Ron Kaplan
> Sent: Wednesday, June 9, 2021 8:01 PM
> To: Larry Masinter <L...@acm.org>
> Cc: Interlisp core <lisp...@googlegroups.com>
> Subject: Re: gather-info, msnoticefile, and dumpdatabase
>
Ron Kaplan
Jul 18, 2021, 2:29:26 PM7/18/21
to Larry Masinter, Interlisp core
I think I fixed some of these issues (although perhaps only in the branch that is awaiting a merge).
I went through and qualified all the conflicting field names.
I found that putting the name of the file in quotes (e.g. (. ANALYZE on (QUOTE RECORD)) fixes that problem, the fact that that particular filename is also a masterscope keyword.
What is still outstanding, I think, is that the CONTAINS table is not constructed in the analysis, Masterscope queries seem to depend on the incore FILECOMS. So the whole-system database file doesn’t contain the information for files that are not loadfrom-ed.
I patched that a little bit by having Masterscope go off to the WHEREIS database, as a last resort. But that doesn’t get the inverse relation (. WHO DOES X CONTAIN).
In terms of the whole-system analysis, the only warning messages are the relatively few misplaced comment-ones. Those probably should also be fixed.
As for DUMPDATABASE, I extended it so that it takes a file as an extra argument, and it puts out reader-environment information so that it can be loaded in any context.
> To view this discussion on the web visit https://groups.google.com/d/msgid/lispcore/048a01d77c01%243f74a6e0%24be5df4a0%24%40acm.org.
I went through and qualified all the conflicting field names.
I found that putting the name of the file in quotes (e.g. (. ANALYZE on (QUOTE RECORD)) fixes that problem, the fact that that particular filename is also a masterscope keyword.
What is still outstanding, I think, is that the CONTAINS table is not constructed in the analysis, Masterscope queries seem to depend on the incore FILECOMS. So the whole-system database file doesn’t contain the information for files that are not loadfrom-ed.
I patched that a little bit by having Masterscope go off to the WHEREIS database, as a last resort. But that doesn’t get the inverse relation (. WHO DOES X CONTAIN).
In terms of the whole-system analysis, the only warning messages are the relatively few misplaced comment-ones. Those probably should also be fixed.
As for DUMPDATABASE, I extended it so that it takes a file as an extra argument, and it puts out reader-environment information so that it can be loaded in any context.
Reply all
Reply to author
Forward
0 new messages