Adding project-level scoping to raco pkg install

[Sage Gerard]

Alright, so after thinking about how to improve Racket package management I decided that I want to add project scoping logic to raco pkg install. I'm not sure --scope-dir is enough because the docs say that the installation is apparently checked before the scoped directories, and there are no lock files. Correct me if I'm wrong.

I've previously made commitments re: a CI dashboard and a syntax highlighter. Those remain, but don't seem to need me right this minute. Meanwhile, this is one of those features that hurts to not have, especially with current limitations in the default package catalog.

https://akkuscm.org says it supports plt-r6rs, so I see two paths: Either implement the diff between that and Racket today in Akku, or add just enough of Akku to a raco pkg install --scope project command. I expect both efforts would be painful, but the latter seems better.

Are there any landmines are in the raco pkg install code that I need to know about? I'd rather not make a bunch of changes with subtle consequences that I'm not seeing. I'd also like to know who would be the best person to review my proposed changes in this area.

~slg

[Matthew Flatt]

Consider viewing "project" scope as "installation" scope, where an
existing installation becomes a wider scope after "installation". Then,
the exiting configuration and scope machinery will match what you need.

If you take that view, then a project gets its own "etc" where you put
a "config.rktd" to configure the project-as-installation. That
"config.rktd" can point to the original installation as the next layer
for various searches: "collects", "lib", and so on.

You can even set `config-tethered-console-bin-dir` and
`config-tethered-gui-bin-dir` to get project-local `racket`, `raco`,
etc., executables that have the right configuration path built in,
instead of having to use something like environment variables to select
a project's configuration.

At least, that's one of the intentions of the existing layers. I didn't
push it further (e.g., to a `raco` command that would set up a project
in this sense) because there was not demand at the time.

At Thu, 30 Apr 2020 19:10:11 +0000, Sage Gerard wrote:
> Alright, so after thinking about how to improve Racket package management I
> decided that I want to add project scoping logic to raco pkg install. I'm not
> sure --scope-dir is enough because the docs say that the installation is
> apparently checked before the scoped directories, and there are no lock files.
> Correct me if I'm wrong.
>
> I've previously made commitments re: a CI dashboard and a syntax highlighter.
> Those remain, but don't seem to need me right this minute. Meanwhile, this is
> one of those features that hurts to not have, especially with current
> limitations in the default package catalog.
>

> [https://akkuscm.org](https://akkuscm.org/) says it supports plt-r6rs, so I

> see two paths: Either implement the diff between that and Racket today in
> Akku, or add just enough of Akku to a raco pkg install --scope project
> command. I expect both efforts would be painful, but the latter seems better.
>
> Are there any landmines are in the raco pkg install code that I need to know
> about? I'd rather not make a bunch of changes with subtle consequences that
> I'm not seeing. I'd also like to know who would be the best person to review
> my proposed changes in this area.
>
> ~slg
>

> --
> You received this message because you are subscribed to the Google Groups
> "Racket Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to racket-dev+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/racket-dev/04pE-yKBt1FzXg9NpgHlRoMEPSp3rIKJMJ
> fzPRd7HBHxyF1gEwxK70y4uFONwvTXUADMdH5_Yz-Xy7fHA7XrbWe6NYYHXvE65gvPZ4H_uNU%3D%40
> sagegerard.com.

[Sage Gerard]

Some of the pieces are falling into place, but I'm still chewing on something.

Let me rephrase my goal with added details: As a library author, I need my users to have ONE command that safely installs a package with collection paths that would collide in a user or (system) installation scope.

The docs on package scopes and your email make it sound like I have to ask my users to create and configure an installation, and make sure no colliding packages exist at user scope. All before installing the package. Can your recipe be done in one step? If not, I think that's where I'd like to contribute.

~slg

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

> > https://akkuscm.org says it supports plt-r6rs, so I

> > see two paths: Either implement the diff between that and Racket today in
> > Akku, or add just enough of Akku to a raco pkg install --scope project
> > command. I expect both efforts would be painful, but the latter seems better.
> > Are there any landmines are in the raco pkg install code that I need to know
> > about? I'd rather not make a bunch of changes with subtle consequences that
> > I'm not seeing. I'd also like to know who would be the best person to review
> > my proposed changes in this area.
> > ~slg
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Racket Developers" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to racket-dev+...@googlegroups.com.
> > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/racket-dev/04pE-yKBt1FzXg9NpgHlRoMEPSp3rIKJMJ
> > fzPRd7HBHxyF1gEwxK70y4uFONwvTXUADMdH5_Yz-Xy7fHA7XrbWe6NYYHXvE65gvPZ4H_uNU%3D%40
> > sagegerard.com.
>
> --
>
> You received this message because you are subscribed to the Google Groups "Racket Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to racket-dev+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/racket-dev/5eab25df.1c69fb81.8f98.25f6SMTPIN_ADDED_MISSING%40gmr-mx.google.com.

[Matthew Flatt]

That's right: there's no one-step operation right now, and a new
one-step tool that sets things up the way you need would be a great
contribution.

To avoid anything currently in user scope, maybe your new tool would
configure the project space with a fresh 'installation-name. After
setting 'pkgs-dir to be the project-specific package directory while
setting 'pkg-search-dir to chain to the original installation's package
directory, it could install some initial packages in the new
"installation" scope --- without affecting "user" scope or the original
"installation" scope. By setting 'lib-dir and 'shared-dir and similar
directories to a project-specific location, any files installed by
packages in the new "installation" scope would not pollute the old
installation. And so on.

I'm not sure of the details, and it's possible that something new is
would still be needed in the collection or package or `setup/dirs`
layers to make the one-step tool work right. It's still possible that
the existing support doesn't do what you want at all.

I'm unclear on whether you want to install packages in project scope
that would collide with things in the original "installation" scope,
but that's tricky at best. Conceptually, it works to install a package
in a narrower scope and shadow one in a wider scope, but only if
everything in the wider scope that depends on the shadowed package is
also shadowed in the narrower scope. If you want to be able to do that,
then probably this isn't the right direction.

> https://groups.google.com/d/msgid/racket-dev/il5M1m5pESWE_CWHdsPHg9WvQ5BhXin9sT
> UJ9KMwaoKzuZldA1874eMaM5py5OFrsd23Az6Kr9mH69YblDfqrGS77flNW3BnwVzQGvygG84%3D%40
> sagegerard.com.

[Robby Findler]

I'm trying to understand what you guys are talking about here and am not sure I'm getting it. Is the idea that a "project" (new term definition) would be a place one can install a package and its dependencies in a way that avoids any conflicts with other pkgs that are already installed (even if they are different versions of those same packages)? So I would have some command to configure my installation so that I would tell it what project I want to work with and then uses of `racket` after that would find requires inside the pkgs that belong to that project?

(Sorry if this is a dumb question; I'm very curious about this.)

Robby

To view this discussion on the web visit https://groups.google.com/d/msgid/racket-dev/5eab440d.1c69fb81.87557.2d50SMTPIN_ADDED_MISSING%40gmr-mx.google.com.

[Sage Gerard]

I'm looking for an experience similar to how NPM, Pipenv, and other package managers work in their respective ecosystems. You described the desired conditions correctly.

Racket's separation of packages and collections complicates that a bit. Sam mentioned on the Slack that this problem could be approached in terms of 'raco link', but it would lead to a large scope of work. I wonder how much that overlaps here.




-------- Original Message --------

To view this discussion on the web visit https://groups.google.com/d/msgid/racket-dev/CAL3TdOOgXhcPV-ppNy%3D%2Be%2B_LU6B-1jd-L1jmUiZT%2B6O9WYCa4g%40mail.gmail.com.

[Robby Findler]

I'm not familiar enough with NPM or Pipenv to fully appreciate what you're saying, but I'm glad it sounds like I have the right basic idea. raco link is, roughly speaking, a layer below raco pkg that is used to implement raco pkg.

Well, at any rate, I'll be curious to see how things progress!

Robby

[Alex Harsanyi]

On Friday, May 1, 2020 at 9:01:41 AM UTC+8, Robby Findler wrote:

I'm trying to understand what you guys are talking about here and am not sure I'm getting it.

 

Is the idea that a "project" (new term definition) would be a place one can install a package and its dependencies in a way that avoids any conflicts with other pkgs that are already installed (even if they are different versions of those same packages)?

The "project" would be the toplevel folder where  the source files for a Racket application (not a package) are located.  Inside this folder, perhaps in a ".racket" sub-folder, the  packages used by the application would be installed.  Than, any `requires` inside this "project" folder will be resolved with packages from the .racket sub-folder first, than the other package scopes would be searched.

Another Racket application would be another project and would have its package dependencies installed in its own .racket subfolder, these packages won't conflict with the packages from the first installation.  If a "package-a" is installed in the first project, trying to use "(require package-a)" from a file in the second project will not find it.

 

So I would have some command to configure my installation so that I would tell it what project I want to work with and then uses of `racket` after that would find requires inside the pkgs that belong to that project?

To use an analogy with GIT, you can `cd` into one repository and run "git push origin" and it will push some commits to some remote location, than `cd` into a different directly and run the same "git push origin" and it will push a different set of commits to some other location.  You don't tell your GIT installation that you switched projects, and you don't register your projects with your GIT installation in the first place, yet git was able to determine that "origin" in the first case means something else from "origin" in the second case.

 

I am also interested in such a feature, and the examples I gave are only an overview on how I would like things work.  It is also similar how NPM (the node package manager) does things.  There are, of course many other things to consider and it might not be a trivial thing to implement.

Alex.

 

[Sage Gerard]

> The "project" would be the toplevel folder where  the source files for a Racket application (not a package) 

Why can it not be a package? I would be very, very interested in finding a way to make this work for packages as well.
-------- Original Message --------

--

You received this message because you are subscribed to the Google Groups "Racket Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to racket-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/racket-dev/edda4402-2203-4c91-b261-cd5f9a65f767%40googlegroups.com.

[Alex Harsanyi]

On Friday, May 1, 2020 at 1:02:04 PM UTC+8, Sage Gerard wrote:

> The "project" would be the toplevel folder where  the source files for a Racket application (not a package) 

Why can it not be a package? I would be very, very interested in finding a way to make this work for packages as well.

If you want this to work for packages as well you will have to deal with the case where two versions of the same package are required in the same racket process. E.g. package A depends on packages B and C, B depends on package D version 1, while C depends on package D version 2.

This complicates things somewhat.

Alex.

[Sage Gerard]

I know this touches on a reason why PLaneT had issues, but I don't understand how this is a problem. If two versions of the same package are in different locations on a filesystem and the module resolver can somehow distinguish the two using collection paths, then how else do they collide?


-------- Original Message --------

--

You received this message because you are subscribed to the Google Groups "Racket Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to racket-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/racket-dev/017572c9-8d38-4899-b4a3-da03acd783ab%40googlegroups.com.

[Sam Tobin-Hochstadt]

I'll try to write out in more detail what the kinds of package
managers Sage is referring to do, to make this clearer for everyone.

The high-level idea is that you have a bunch of directories on your
system(Alex's analogy to git is a good one), and in each of them, your
run appropriate commands and you see things set up as if the other
directories you've worked with didn't exist. This includes both
executing code (like running `racket foo.rkt`) and managing packages
(like `raco pkg install`).

There are basically two approaches in wide use here, depending on
whether the underlying system was developed after this kind of package
management approach became popular (npm, cargo, stack) or before
(virtualenv for python and others).

Approach 1 involves mostly working using the package manager tool
(such as npm), but also setting up the semantics of the module system
to look in the local directory.

Approach 2 involves setting environment variables and command line
arguments, and wrapping the underlying command (such as the python
binary) to trick it into looking for files in the right place.

In both approaches, installing libraries "globally" is not
recommended, and likely reserved for command line tools that aren't
depended upon as libraries, or for core system libraries that can't
change independently of the compiler.

One additional detail -- often performing package operations will
automatically record the exact versions of everything that was used in
that directory (with data similar to a pkg catalog), so that a the
same code can be installed with the same dependent packages in a
different situation.

I think supporting this mode of working would be very useful for
Racket, but there are definitely some challenges. The biggest one
relates to what Matthew mentioned, with globally-installed packages
shadowed by local-scope packages. This would be particularly hard for
Racket, where lots of tools we want to install once globally (like
DrRacket) would pull in packages that would also be needed locally.
Additionally, how DrRacket would work in this mode would be tricky,
since DrRacket's architecture relies on sharing of lower-level modules
between the program and DrRacket itself.

With that said, I think the approach Matthew outlined would take a big
step, and would let us find out if we needed more steps, and what they
might be.

Sam

On Thu, Apr 30, 2020 at 11:59 PM Robby Findler
<ro...@cs.northwestern.edu> wrote:
>
> I'm not familiar enough with NPM or Pipenv to fully appreciate what you're saying, but I'm glad it sounds like I have the right basic idea. raco link is, roughly speaking, a layer below raco pkg that is used to implement raco pkg.
>
> Well, at any rate, I'll be curious to see how things progress!
>

> Robby
>
>

> On Thu, Apr 30, 2020 at 8:18 PM Sage Gerard <sa...@sagegerard.com> wrote:
>>

> To view this discussion on the web visit https://groups.google.com/d/msgid/racket-dev/CAL3TdOMwspLvuha7FoOm_RApfY-YGq%2BsB2625%3DLxBBJ0vhiYng%40mail.gmail.com.

[Robby Findler]

Thanks! Although I cannot really see the full implications, the basic idea sounds really great.

DrRacket does share some things, but tries to minimize the sharing; if it were possible to share less, that'd be great, but maybe that'll end up falling under the same category as reasons you'd install something globally. But certainly it would be important for the system to support the ability to change which "project" you're in without having to start a new command-line racket binary somehow.

Robby

To view this discussion on the web visit https://groups.google.com/d/msgid/racket-dev/CAK%3DHD%2BYR_x4nuGV78-TB0dDzbAF%3D6hZDWQ%3DkbY9Pn4QqBJawcw%40mail.gmail.com.

[Simon Schlee]

I find this discussion interesting, it caused me to search around for links, blogposts, papers and examples about package

 management. The things I found so far are accessible here (a online collaborative text editor), but feel free to move it to some other tool, wiki etc.:

https://cryptpad.fr/pad/#/2/pad/edit/YBPAp3HwnjdxbaDIVWaEcfrV/

Maybe we can collect links to papers that could be relevant.

Probably should be converted to a github repo / wiki at some point?

For packaging standalone applications this issue might be partially related to this discussion:
"Easy to use and scriptable cross platform distribution support"

https://github.com/racket/rhombus-brainstorming/issues/113

I am interested in helping with any potential efforts towards new or extended tools related to this.

Simon

To view this discussion on the web visit https://groups.google.com/d/msgid/racket-dev/CAL3TdOP0T693W2Avw3R-rgmAUPbdovJcVn2zJRqx0Cb-gwD66w%40mail.gmail.com.

[Sorawee Porncharoenwase]

There are basically two approaches in wide use here, depending on
whether the underlying system was developed after this kind of package
management approach became popular (npm, cargo, stack) or before
(virtualenv for python and others).

`stack` (Haskell) is a bit different from `npm` (JS): it installs a new compiler for each project. I can't imagine how this approach could work with tools like DrRacket.

I know this touches on a reason why PLaneT had issues, but I don't understand how this is a problem. If two versions of the same package are in different locations on a filesystem and the module resolver can somehow distinguish the two using collection paths, then how else do they collide?

IIUC, `npm` solves this problem by doing exactly this, but it is pretty naive. A solver based approach can be more optimized: http://www.well-typed.com/blog/2015/03/qualified-goals/.

[Sage Gerard]

Thanks for the link. Is there a way to use the solver-based approach that's easier than involving a tethered installation or a custom launcher? On my end it looks like any different approach to packages would need a different module name resolver too.

~slg

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

--

You received this message because you are subscribed to the Google Groups "Racket Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to racket-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/racket-dev/CADcuegtP%2BKEDz4rPQV6uU%2B2yMofizwe12Tb0MX1B24zWGjHE-w%40mail.gmail.com.

[Matthew Flatt]

FWIW:

Changing the search path for modules isn't just about changing the
`current-module-name-resolver` parameter. Paths and search rules end up
getting used in subtly different ways by different tools, such as
`racket` versus `raco setup` versus `raco exe`. The module name
resolver is more of a hook than the definition of resolution. For
example, `raco exe` finds modules using its own search implementation,
but it sets the module name resolver in the generated executable; that
resolver to connects the modules found "statically" when building the
executable to the modules demanded dynamically at run time.

One of the big shifts from Planet to the new package system was to
remove the process of package resolution from the module search path.
Module search is based on collections, and package management installs
collections.

There are drawbacks to separating module search from package
resolution. I just wanted to note that we've tried both, and that
changing search rules can be a bigger and more invasive change than it
appears at first.

> zwe12Tb0MX1B24zWGjHE-w%40mail.gmail.com](https://groups.google.com/d/msgid/rack
> et-dev/CADcuegtP%2BKEDz4rPQV6uU%2B2yMofizwe12Tb0MX1B24zWGjHE-w%40mail.gmail.com
> ?utm_medium=email&utm_source=footer).

>
> --
> You received this message because you are subscribed to the Google Groups
> "Racket Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to racket-dev+...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/racket-dev/4S48b8dOz7vS_k2y1cbrBocbl0IHRsLV9u
> 24M-wSw31KL9bpBZ5Pi-9VowKZIoJSP6MlLJ9UFwUpqLXab5IoypbFf0lbmV-iQYYXUv_AArE%3D%40
> sagegerard.com.

[Sage Gerard]

Thanks for this, it really helps.

I'm not sure what improvements can be made that A) wouldn't repeat the problems encountered in PLaneT and B) gives users an "easy" way to deal with breaking changes in collection names beyond what the Package Management FAQ suggests.

I remember Sam mentioning `raco link` in Slack, but based on your emails it sounds like NOTHING can fundamentally change without a tethered installation or custom distribution (e.g. a collects-like directory relative to a different executable). Is that correct? I wasn't sure if you were hinting at that in your first email.

If that's the case, then I'd probably double down on my first reply: It almost seems like giving users an easy way to generate tethered installations would open up more opportunities than trying to make the default installation behave a certain way around packages, collections, and modules.


~slg

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

> To view this discussion on the web visit https://groups.google.com/d/msgid/racket-dev/20200521085037.1f0%40sirmail.smtp.cs.utah.edu.

[Matthew Flatt]

At Thu, 21 May 2020 15:39:30 +0000, Sage Gerard wrote:
> I'm not sure what improvements can be made that A) wouldn't repeat the
> problems encountered in PLaneT and B) gives users an "easy" way to deal with
> breaking changes in collection names beyond what the Package Management FAQ
> suggests.

Well, I hope you take my comments in the spirit of "advice that may be
wrong". When multiple things change between A and B, saying which
changes mattered ends up being a matter of opinion and interpretation.

> I remember Sam mentioning `raco link` in Slack, but based on your
> emails it sounds like NOTHING can fundamentally change without a
> tethered installation or custom distribution (e.g. a collects-like
> directory relative to a different executable). Is that correct? I
> wasn't sure if you were hinting at that in your first email.

You can do many things with just environment variables (e.g., to pick a
configuration directory). A tethered environment solves the problem
that environment variables are too widely scoped; if your Racket
program is meant to start a separate process that runs a different
Racket, for example, the environment variables affects the second
Racket --- unless the original program goes out of its way to remove
them.

> If that's the case, then I'd probably double down on my first reply:
> It almost seems like giving users an easy way to generate tethered
> installations would open up more opportunities than trying to make
> the default installation behave a certain way around packages,
> collections, and modules.

I still agree with this.

[Sage Gerard]

Reviving this thread. I spent the last couple of months on a package manager that operates at a broader scope than `raco pkg`. It's not done because as you all know, package managers are hard.

The source is at https://github.com/zyrolasting/zcpkg. Note that there's a branch dedicated to using only minimal Racket, and I'll be leaning on that once I learn how to properly set up TCP+TLS.

In the context of this thread, there's a point where a Racket developer depends less on Racket packages and more on specific Racket installations. The name `zcpkg` stands for "Zero-Collection Packages", since it explores extending Racket where packages don't modify the running installation. Obviously this entails more verbose specifications and require specs.

After some reading I found that my goals overlapped with Nix and Guix. In this light, all I'm really doing is moving the boundary for the "unit of exchange" (Sam Boyer's works [1]) to encompass a Racket installation and non-Racket resources. My hope is that will address the topic discussed in this thread, since you can install a Racket package that includes the means by which Racket runs to support that package.

It also acts as a nice use case for reader extensions, since it does not seem like a huge leap from my current state to adapt setup/infotab to what Guile and Nix's language are doing. Jury is still out... I could be embarrassingly wrong.

The HN thread for Boyer's article is also pretty valuable since it has people articulating pain points, and that's coloring my approach. [2]

I'm open to increasing interoperability with `raco pkg`, but I'm honestly unsure if that can be done easily. I know this isn't a PLT-sanctioned approach, but I'd like to know if there's any warmth to the direction I'm headed here.

[1]: https://medium.com/@sdboyer/so-you-want-to-write-a-package-manager-4ae9c17d9527#.740o43vxi
[2]: https://news.ycombinator.com/item?id=11088125


~slg

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

> ---------------------------

>
> You received this message because you are subscribed to the Google Groups "Racket Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to racket-dev+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/racket-dev/20200521095040.19d%40sirmail.smtp.cs.utah.edu.

[Matthias Felleisen]

On Jul 25, 2020, at 5:36 PM, Sage Gerard <sa...@sagegerard.com> wrote:

I know this isn't a PLT-sanctioned approach, but I'd like to know if there's any warmth to the direction I'm headed here.

There is no such thing as “PLT-sanctioned”. If you can improve Racket, go for it. If you can construct tools that are better suited for the working Racket programmer, just do it. Racket is out there for everyone who wants to build a better programming context. We started it and we will not be jealous if someone does better. That’s what it’s here for. 

(An analogy that may not work. My proudest achievement is to hire a young person as an assistant professor who will one day achieve much more than I have.)  

— Matthias

[Matthew Flatt]

At Sat, 25 Jul 2020 21:36:20 +0000, Sage Gerard wrote:
> The source is at https://github.com/zyrolasting/zcpkg.

Thanks for a pointer to the implementation! Trying things out --- and,
to a lesser degree, looking at the source --- answered a lot of my
initial questions.

While I was able to run `zcpkg serve`, I wasn't able to set things up
so that a request via `zcpkg -E ... install ...` downloaded from it. It
may be because I'm unclear on how configurations work. Can you provide
more hints on how to set that up?


I'm unclear whether I'm using/understanding cross-package imports
correctly. If I have a package "alpha" that depends on a package
"beta", where "a.rkt" in "alpha" uses "b.rkt" in "beta", and assuming
that "beta" is at "draft:0", it seems that right now I have to write
this in "a.rkt":

(require "../../../../mflatt/beta/draft/0/b.rkt")

Is that the intent?

I can imagine that the intent is instead to set up links so that

(require "../mflatt/beta/draft/b.rkt")

or maybe even

(require "../mflatt/beta/b.rkt")

would work. Meanwhile, dependency information for the "alpha" package
could specify "draft" and maybe even "draft:0".

[FWIW: Experience with PLaneT was that if you allow dependencies
without an edition, that shorthand will be used often, and then
breaking changes for a new edition turn out to be a problem after all.
But, if I guess right about the intent with `require`, a difference
was that PLaneT dependencies were always encoded in a `require`
reference instead of in separate package metadata.]

> I'm open to increasing interoperability with `raco pkg`, but I'm honestly
> unsure if that can be done easily. I know this isn't a PLT-sanctioned
> approach, but I'd like to know if there's any warmth to the direction I'm
> headed here.

As Matthias says, I'm skeptical of "PLT-sactioned". "Interesting to
people who know the current internals, and enough that they'd help make
sure it fits together" is a sensible question, though I can only answer
for myself.

I think this is an interesting direction.

Interoperability with `raco pkg` might mean that code could be used
either in `raco pkg` mode or `zcpkg` mode. If so, I can imagine a new
`require` subform that is

* treated like a "../" path when resolved relative to a filesystem
path (as it would appear in a `zcpkg` install), and

* treated like a collection-based path when resolved relative to a
collection-based path (as it would appear in a `raco pkg` install).

Things that manipulate modules paths for different purposes would all
have to change to deal with the new form, but I think it might work.
Crucially, your design avoids search paths, which cause all sorts of
trouble. In the implementation that I imagine, a `zcpkg`-installed
package would be able to see `raco pkg`-installed packages, but not
vice versa, which I think it what you have in mind.

But my guesses depend in part on whether I've understood the intended
way of referencing modules across `zcpkg` packages.


Matthew

[Sage Gerard]

Apologies for the "PLT-sanctioned" comment. I came off
differently than intended.

> While I was able to run `zcpkg serve`, I wasn't able to set things up so that a request via `zcpkg -E ... install ...` downloaded from it.
> It may be because I'm unclear on how configurations work.

Your attempt would have worked. The server just isn't done.
I didn't emphasize it much in the README, but I ended
up taking a detour to work from minimal Racket. That led to
building the service from the transport layer. While
I deal with that, I'm afraid the only way to preview
functionality now is by installing local directories.

> I'm unclear whether I'm using/understanding cross-package imports correctly.

A conventional "zcpkg-deps" directory is made available in
installed packages as defined in https://github.com/zyrolasting/zcpkg/blob/master/file.rkt#L114
You're looking for something like:

(require (file "zcpkg-deps/example.com/package/module.rkt")).

I wouldn't want to ask someone to navigate a directory structure
outside of their package boundary. Obviously a require transformer
would be nice, but at the level I'm targeting I have the chicken-and-egg
problem re: where that binding comes from. For now, this is the added
verbosity I mentioned. Currently I assume that a dependent package
only needs one version of a package, which is why the path above
does not include version information. I haven't yet
decided if this should change.

> Interoperability with `raco pkg` might mean that code could be used
> either in `raco pkg` mode or `zcpkg` mode.

I visualized the project as adding casework: Racket packages define at
least one collection, so what changes with a zero-collection package?
Well, everything, so it felt more like `raco pkg` having an "advanced" mode.
It seemed wise to keep my code isolated while I figured things out.

After reading Sam Boyer's article, it might be better to say that zcpkg
is a PDM and raco pkg is an LDM. If `raco pkg` could operate such that
PDM, LDM, and SDM were treated as modes, I suppose that would work just as
well. It would certainly give a minimal Racket installation more powers
over userspace, but I imagine the CLI would be fiendishly complicated.

~slg

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

> - treated like a "../" path when resolved relative to a filesystem

> path (as it would appear in a `zcpkg` install), and
>

> - treated like a collection-based path when resolved relative to a

> collection-based path (as it would appear in a `raco pkg` install).
>
> Things that manipulate modules paths for different purposes would all
> have to change to deal with the new form, but I think it might work.
> Crucially, your design avoids search paths, which cause all sorts of
> trouble. In the implementation that I imagine, a `zcpkg`-installed
> package would be able to see `raco pkg`-installed packages, but not
> vice versa, which I think it what you have in mind.
>
> But my guesses depend in part on whether I've understood the intended
> way of referencing modules across `zcpkg` packages.
>
> Matthew
>

> --
> You received this message because you are subscribed to the Google Groups "Racket Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to racket-dev+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/racket-dev/20200725193756.13a%40sirmail.smtp.cs.utah.edu.
>

[Matthew Flatt]

At Sun, 26 Jul 2020 02:58:26 +0000, Sage Gerard wrote:
> You're looking for something like:
>
> (require (file "zcpkg-deps/example.com/package/module.rkt")).

Ah, got it. In my example, I wanted

(require "zcpkg-deps/mflatt/beta/b.rkt")

So, I understood the intent, but I missed "zcpkg-deps" instead of "..".

> Currently I assume that a dependent package
> only needs one version of a package, which is why the path above
> does not include version information. I haven't yet
> decided if this should change.

Makes sense.

> I visualized the project as adding casework: Racket packages define at
> least one collection, so what changes with a zero-collection package?
> Well, everything, so it felt more like `raco pkg` having an "advanced" mode.
> It seemed wise to keep my code isolated while I figured things out.

Yes. Even if we set up some kind of interoperability with `raco pkg`, I
can imagine that the implementations of the two modes staying almost
completely separate.

[reordered:]

> Obviously a require transformer would be nice, but at the level I'm
> targeting I have the chicken-and-egg problem re: where that binding
> comes from.

That makes sense, too.

To be clear, the possibility I have in mind for letting code migrate
between `raco pkg` and `zcpkg` modes involves an even more pervasive
change --- modifying the grammar of module path --- not just adding a
require transformer. But that's probably getting too far ahead of
things right now.

Matthew

[Sage Gerard]

zcpkg is now in an alpha state. The server and client are operational, but the docs are not done. I'm preparing a short demo video to show the current status to make up for that.

https://github.com/zyrolasting/zcpkg/releases/tag/alpha

A couple of neat things are possible, but I haven't done them yet:

- A Linux distribution can be defined in terms of zcpkg commands.
- zcpkg can install itself.

~slg

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

>
> You received this message because you are subscribed to the Google Groups "Racket Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to racket-dev+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/racket-dev/20200726070204.31%40sirmail.smtp.cs.utah.edu.