make & --clone installed pkgs

17 views
Skip to first unread message

Robby Findler

unread,
Feb 17, 2015, 9:54:53 AM2/17/15
to d...@racket-lang.org
Sam and I have run into a situation where `make` fails because we've
set up clone pkgs and made local modifications in a way that makes the
git commands fail [*].

My guess is that the right thing to do is for me to know about these
pkgs and do something special when running make. I'm thinking that I
could let make finish the initial steps until it gets to the step
where it updates the pkgs and then do the update step myself and then
run `make as-is`. But the problem with this is that I don't see what
command I can run that will update all of the pkgs except the
problematic ones. Like I could imagine a `raco pkg update
--all-except-clones` or something, but that feels a bit strange as
there could be other development modes that would also run into
similar problems. Maybe `raco pkg update
--all-things-from-this-catalog
<the-catalog-I-currently-get-main-distribution-from>` or something
along those lines is the way to go? In general, it seems right for me
to run some commands whose complications are roughly proportional to
the number of pkgs that I have installed as clones (and where I'm
actively developing them) but not to run some commands that require me
to do something special for each pkg that is installed.

Any ideas? Or am I just missing the right command?

Thanks,
Robby

[*] In my case, in case this suggests a better/different approach to a
resolution: the `raco pkg update` step eventually gets to this git
command:

git merge --ff-only <sha1>

where the <sha1> is the checksum from the pkg server, I believe. In my
case, this is a different branch than is currently checked in my
clone'd pkg and so the git merge command fails (and that command
failing seems like the right behavior for the setup I'd like to be
able to use).

Sam Tobin-Hochstadt

unread,
Feb 17, 2015, 10:39:05 AM2/17/15
to Robby Findler, d...@racket-lang.org
I think there are two seperable issues here:

1. Can we make `raco pkg update -a` better/more robust in this case?

2. Should `make` run `raco pkg update -a`?

In reverse order:

- I think `make`, by default, shouldn't update anything, and that we
should have a different Makefile target which updates things when
people want that explicitly. The current behavior is especially
problematic because it updates things that aren't in
"main-distribution", meaning that it's making potentially arbitrary
breaking changes to software on your computer (not just to core
Racket).

This could be more inconvenient for someone working widely on core
packages, but if they wanted the current behavior it would be just
`make update` (or some other name) instead of `make`. As someone who
does work on a lot of core packages, I'd prefer greater explicitness.

- I think `raco pkg update p` where `p` is a cloned package should
only do anything if (a) the currently-checked-out branch is the one in
the pkg source and (b) the `git merge --ff-only` command would
succeed. Otherwise, I think it should just print a message and leave
the repository as it is. I think that's what I wanted all the times
that this operation has failed in my experience.

Sam
> --
> You received this message because you are subscribed to the Google Groups "Racket Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to racket-dev+...@googlegroups.com.
> To post to this group, send email to racke...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/racket-dev/CAL3TdONOjL3_y8U_A1ZUoz_1Z%3DE3HjVV8by9e%2B2dS-W2mc51pg%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

Matthew Flatt

unread,
Feb 17, 2015, 1:03:27 PM2/17/15
to Sam Tobin-Hochstadt, Robby Findler, d...@racket-lang.org
Continuing in reverse order:

- My sense is that the switch to `make` so that it updates packages,
which was a result of

http://lists.racket-lang.org/users/archive/2015-January/065345.html

has been a good change for most people most of the time.

The `as-is` target is currently available for building without updating
packages. I think it would make sense to introduce `make
as-is-the-default` to effectively change the default `make` target to
be `make as-is`. (And there would be `make with-update` to get the
current behavior of `make` after `make as-is-the-default`, plus `make
with-update-the-default` for completeness.)


- It makes sense to me to add a `--pull` option to `raco pkg update`
for specifying the behavior of the `git pull` step for linked clones. I
imagine that the default would be `--pull ff-only` for the current
behavior, while `--pull try` would ignore fast-forward failures. I
think I'd also like `--pull rebase`.

I'm skeptical that making `--pull try` the default would be a good
idea. People have trouble seeing warnings.


Would those changes work?
> https://groups.google.com/d/msgid/racket-dev/CAK%3DHD%2BbYrWHAxWssmvXFA%2BA0tco
> 2RDAXwORQj_iwOq%3DpYXzZOA%40mail.gmail.com.

Robby Findler

unread,
Feb 17, 2015, 1:07:49 PM2/17/15
to Matthew Flatt, Sam Tobin-Hochstadt, d...@racket-lang.org
That would work great for me. And I think that I recall that "make
base" is the target that I would use to build enough to be able to run
`raco pkg update --all --auto --pull try` and then finish off with
`raco setup`, right?

Robby

Sam Tobin-Hochstadt

unread,
Feb 17, 2015, 2:13:15 PM2/17/15
to Matthew Flatt, Robby Findler, d...@racket-lang.org
On Tue, Feb 17, 2015 at 1:03 PM, Matthew Flatt <mfl...@cs.utah.edu> wrote:
> Continuing in reverse order:
>
> - My sense is that the switch to `make` so that it updates packages,
> which was a result of
>
> http://lists.racket-lang.org/users/archive/2015-January/065345.html
>
> has been a good change for most people most of the time.
>
> The `as-is` target is currently available for building without updating
> packages. I think it would make sense to introduce `make
> as-is-the-default` to effectively change the default `make` target to
> be `make as-is`. (And there would be `make with-update` to get the
> current behavior of `make` after `make as-is-the-default`, plus `make
> with-update-the-default` for completeness.)

I continue to think that updating pkgs automatically is the wrong
default, but regardless of where we end up on the default I don't know
if adding yet more make targets is a good idea. I can switch to using
`make as-is`.

Regardless of that, though, I think we should switch to updating only
"main-distribution" (and perhaps "main-distribution-tests"). I doubt
people expect `make` in the Racket source tree to update their
software somewhere else on their machine -- I certainly would be very
unpleasantly surprised if that happened to me when rebuilding some
other language I had installed.

> - It makes sense to me to add a `--pull` option to `raco pkg update`
> for specifying the behavior of the `git pull` step for linked clones. I
> imagine that the default would be `--pull ff-only` for the current
> behavior, while `--pull try` would ignore fast-forward failures. I
> think I'd also like `--pull rebase`.
>
> I'm skeptical that making `--pull try` the default would be a good
> idea. People have trouble seeing warnings.

I think there's an important difference between ignoring failures and
not trying to merge in situations where it doesn't make sense. For
example, if some other branch is checked out, I don't think skipping
updating is "ignoring failure". I guess I would label my preference
`--pull conservative` and that even if you miss the warning, it
wouldn't be so bad -- you almost certainly didn't want to change the
package in that case.

Relatedly, perhaps `raco setup` and `raco pkg update` have too much
output currently, making it harder to see warnings?

Sam


>
> Would those changes work?
>
>
> At Tue, 17 Feb 2015 10:38:43 -0500, Sam Tobin-Hochstadt wrote:
>> https://groups.google.com/d/msgid/racket-dev/CAK%3DHD%2BbYrWHAxWssmvXFA%2BA0tco
>> 2RDAXwORQj_iwOq%3DpYXzZOA%40mail.gmail.com.

Matthias Felleisen

unread,
Feb 17, 2015, 5:40:33 PM2/17/15
to Sam Tobin-Hochstadt, Matthew Flatt, Robby Findler, d...@racket-lang.org

On Feb 17, 2015, at 2:12 PM, Sam Tobin-Hochstadt <sa...@cs.indiana.edu> wrote:

> Regardless of that, though, I think we should switch to updating only
> "main-distribution" (and perhaps "main-distribution-tests"). I doubt
> people expect `make` in the Racket source tree to update their
> software somewhere else on their machine -- I certainly would be very
> unpleasantly surprised if that happened to me when rebuilding some
> other language I had installed.


Speaking as the user I am, I really like it that make updates
my extra-pkgs.

Matthew Flatt

unread,
Feb 17, 2015, 6:41:34 PM2/17/15
to Matthias Felleisen, Sam Tobin-Hochstadt, Robby Findler, d...@racket-lang.org
At Tue, 17 Feb 2015 14:12:54 -0500, Sam Tobin-Hochstadt wrote:
> Regardless of that, though, I think we should switch to updating only
> "main-distribution" (and perhaps "main-distribution-tests"). I doubt
> people expect `make` in the Racket source tree to update their
> software somewhere else on their machine -- I certainly would be very
> unpleasantly surprised if that happened to me when rebuilding some
> other language I had installed.

I have trouble seeing the comparison to other software. I don't know of
anything else with an in-place build option, where you can install or
link additional packages in place, and you can also pull core updates,
and somehow it all works. The examples I have seen are all more like
`make unix-style`, where there's a clear separation of the source and
the installed form. In that case, the issues we're looking at here
don't come up.

Does another system have a Racket-like in-place option (that works
better)?


At Tue, 17 Feb 2015 17:40:36 -0500, Matthias Felleisen wrote:
> Speaking as the user I am, I really like it that make updates
> my extra-pkgs.

Package scope provides one a way to get these different behaviors. The
current `make` updates only packages that are in installation scope,
and it also sets installation scope to be the default, so that's why
`make` tends to update everything that is installed. Maybe Sam should
install additional packages in user scope, and then `make` won't try to
update them.

Another possibility is that the packages installed by `make` should go
into a scope that's even deeper than installation scope. That's how the
pre-split organization worked, and I sometimes think that we should go
back to that. But it's more complex, and it would work less well for
Matthias and others who benefit from the current `make` behavior.

Overall, my sense is that the current defaults work right for people
with the least expertise, but we can refine our tools to provide
options to people with more expertise.

Sam Tobin-Hochstadt

unread,
Feb 17, 2015, 8:00:00 PM2/17/15
to Matthew Flatt, Matthias Felleisen, Robby Findler, d...@racket-lang.org
On Tue, Feb 17, 2015 at 6:41 PM, Matthew Flatt <mfl...@cs.utah.edu> wrote:
> At Tue, 17 Feb 2015 14:12:54 -0500, Sam Tobin-Hochstadt wrote:
>> Regardless of that, though, I think we should switch to updating only
>> "main-distribution" (and perhaps "main-distribution-tests"). I doubt
>> people expect `make` in the Racket source tree to update their
>> software somewhere else on their machine -- I certainly would be very
>> unpleasantly surprised if that happened to me when rebuilding some
>> other language I had installed.
>
> I have trouble seeing the comparison to other software. I don't know of
> anything else with an in-place build option, where you can install or
> link additional packages in place, and you can also pull core updates,
> and somehow it all works. The examples I have seen are all more like
> `make unix-style`, where there's a clear separation of the source and
> the installed form. In that case, the issues we're looking at here
> don't come up.
>
> Does another system have a Racket-like in-place option (that works
> better)?

I haven't used it, but GHC has an in-place build option where you can
install packages; see [1]. From their wiki, it looks like `git pull &&
make` will update both the compiler and the "boot libraries" (the ones
used by the compiler), plus any that you've listed as "extra"
explicitly [2], but not any other packages. However, it doesn't seem
like they expect people to use this as the "standard" way to build the
compiler the way we do for Racket.

[1] https://ghc.haskell.org/trac/ghc/wiki/Debugging/InstallingPackagesInplace
[2] https://ghc.haskell.org/trac/ghc/wiki/Commentary/Libraries

I think the closer analogy is to what other software does when you run
both `make` and `make install` analagous to Racket's unix-style
installation, since that's the recommended way of building (eg
Python), just as in-place is the recommended way of building Racket
from source. I don't think any of those systems update packages when
running `make install`.

> At Tue, 17 Feb 2015 17:40:36 -0500, Matthias Felleisen wrote:
>> Speaking as the user I am, I really like it that make updates
>> my extra-pkgs.
>
> Package scope provides one a way to get these different behaviors. The
> current `make` updates only packages that are in installation scope,
> and it also sets installation scope to be the default, so that's why
> `make` tends to update everything that is installed. Maybe Sam should
> install additional packages in user scope, and then `make` won't try to
> update them.

I expect that the packages that update for Matthias on `make` are
packages in "main-distribution", and thus just switching to `raco pkg
update --update-deps main-distribution` would still make him happy
(and be more like the old `git pull && make`).

As an aside, the reason I don't install in user scope is that I switch
between Racket implementations regularly, which would lead to
out-of-date zo errors for all my user packages (instead, I get
multiple copies of the packages).

> Another possibility is that the packages installed by `make` should go
> into a scope that's even deeper than installation scope. That's how the
> pre-split organization worked, and I sometimes think that we should go
> back to that. But it's more complex, and it would work less well for
> Matthias and others who benefit from the current `make` behavior.

Whether that changes Matthias' work flow depends on what packages he's
relying on `make` updating, right?

Sam

Matthias Felleisen

unread,
Feb 17, 2015, 8:50:03 PM2/17/15
to Sam Tobin-Hochstadt, Matthew Flatt, Robby Findler, d...@racket-lang.org

On Feb 17, 2015, at 7:59 PM, Sam Tobin-Hochstadt wrote:

> I expect that the packages that update for Matthias on `make` are
> packages in "main-distribution",


Personally, I have used the 'same' one-line command
going back to csv through svn and now git (_update).

When I write "Speaking as the user I am .." I am
thinking of people like me. Then again, you're
probably saying that I am the bottom element of
the lattice of make/update knowledge, so never mind
what I write.

-- Matthias

Sam Tobin-Hochstadt

unread,
Feb 17, 2015, 9:11:36 PM2/17/15
to Matthias Felleisen, Matthew Flatt, Robby Findler, d...@racket-lang.org
On Tue, Feb 17, 2015 at 8:49 PM, Matthias Felleisen
<matt...@ccs.neu.edu> wrote:
>
> On Feb 17, 2015, at 7:59 PM, Sam Tobin-Hochstadt wrote:
>
>> I expect that the packages that update for Matthias on `make` are
>> packages in "main-distribution",
>
>
> Personally, I have used the 'same' one-line command
> going back to csv through svn and now git (_update).
>
> When I write "Speaking as the user I am .." I am
> thinking of people like me. Then again, you're
> probably saying that I am the bottom element of
> the lattice of make/update knowledge, so never mind
> what I write.

No, what I mean is that many (all?) of the packages you're thinking of
are ones, like "2htdp", that are part of "main-distribution", and thus
you're in a different position than someone who develops packages that
are not part of the distribution.

I certainly don't think you're the bottom element in that lattice --
after all, you've written a shell script to automate your work.

Sam

Matthew Flatt

unread,
Feb 18, 2015, 7:52:07 AM2/18/15
to Sam Tobin-Hochstadt, Matthias Felleisen, Robby Findler, d...@racket-lang.org
At Tue, 17 Feb 2015 19:59:38 -0500, Sam Tobin-Hochstadt wrote:
> On Tue, Feb 17, 2015 at 6:41 PM, Matthew Flatt <mfl...@cs.utah.edu> wrote:
> > At Tue, 17 Feb 2015 14:12:54 -0500, Sam Tobin-Hochstadt wrote:
> > Does another system have a Racket-like in-place option (that works
> > better)?
>
> I haven't used it, but GHC has an in-place build option where you can
> install packages; see [1].

I don't have a lot of experience with GHC, but I have talked to some
GHC and Cabal developers about how they work with the package system.
Those discussions did not lead me to believe that they have an
especially smooth system for working with packages and updates in-place
--- and, in particular, that it's not the way they normally work. I'd
be happy to hear more from someone who routinely works that way with
GHC, though.

> I think the closer analogy is to what other software does when you run
> both `make` and `make install` analagous to Racket's unix-style
> installation, since that's the recommended way of building (eg
> Python), just as in-place is the recommended way of building Racket
> from source. I don't think any of those systems update packages when
> running `make install`.

I'm not sure what you're getting at here. We seem to agree that the
usual `make` plus `make install` is like Racket's `make unix-style`,
neither of which updates packages (other than the ones the makefile
knows about).


> > At Tue, 17 Feb 2015 17:40:36 -0500, Matthias Felleisen wrote:
> >> Speaking as the user I am, I really like it that make updates
> >> my extra-pkgs.
> >
> > Package scope provides one a way to get these different behaviors. The
> > current `make` updates only packages that are in installation scope,
> > and it also sets installation scope to be the default, so that's why
> > `make` tends to update everything that is installed. Maybe Sam should
> > install additional packages in user scope, and then `make` won't try to
> > update them.
>
> I expect that the packages that update for Matthias on `make` are
> packages in "main-distribution"

Ah, no. I've helped Matthias when problems break his installation, I've
noticed that he installs packages not in "main-distribution" (e.g.,
"marketplace"), and I believe he really does want those updated.

I had that context in mind but didn't think to spell it out as I should
have.


> As an aside, the reason I don't install in user scope is that I switch
> between Racket implementations regularly, which would lead to
> out-of-date zo errors for all my user packages (instead, I get
> multiple copies of the packages).

You can give each installation a different name (using `raco pkg config
--set name ...`) to avoid the collision. That would be an extra step in
setting up each new installation, though.


I don't have a strong opinion on whether `make` should update packages
outside of "main-distribution", but the feedback I'm getting is

* Sam doesn't think they should be updated --- but he also doesn't
want packages in "main-distribution" updated, so he's going to use
`make as-is`.

* Everyone else who has spoken up seems to prefer an updating `make`,
so far.

Sam Tobin-Hochstadt

unread,
Feb 18, 2015, 8:01:37 AM2/18/15
to Matthew Flatt, Matthias Felleisen, Robby Findler, d...@racket-lang.org

That seems like a fair summary and since my preference is clearly the minority one, I'm happy to stick with 'make as-is'. The new mode for pulling updates will help, as well.

Sam


On Wed, Feb 18, 2015, 7:52 AM Matthew Flatt <mfl...@cs.utah.edu> wrote:
At Tue, 17 Feb 2015 19:59:38 -0500, Sam Tobin-Hochstadt wrote:
> On Tue, Feb 17, 2015 at 6:41 PM, Matthew Flatt <mfl...@cs.utah.edu> wrote:
> > At Tue, 17 Feb 2015 14:12:54 -0500, Sam Tobin-Hochstadt wrote:
> > Does another system have a Racket-like in-place option (that works
> > better)?
>
> I haven't used it, but GHC has an in-place build option where you can
> install packages; see [1].

I don't have a lot of experience with GHC, but I have talked to some
GHC and Cabal developers about how they work with the package system.
Those discussions did not lead me to believe that they have an
especially smooth system for working with packages and updates in-place
--- and, in particular, that it's not the way they normally work. I'd
be happy to hear more from someone who routinely works that way with
GHC, though.

> I think the closer analogy is to what other software does when you run
> both `make` and `make install` analagous to Racket's unix-style
> installation, since that's the recommended way of building (eg
> Python), just as in-place is the recommended way of building Racket
> from source. I don't think any of those systems update packages when
> running `make install`.

I'm not sure what you're getting at here. We seem to agree that the
usual `make` plus `make install` is like Racket's `make unix-style`,
neither of which updates packages (other than the ones the makefile
knows about).


> > At Tue, 17 Feb 2015 17:40:36 -0500, Matthias Felleisen wrote:
> >> Speaking as the user I am, I really like it that make updates
> >> my extra-pkgs.
> >
> > Package scope provides one a way to get these different behaviors. The
> > current `make` updates only packages that are in installation scope,
> > and it also sets installation scope to be the default, so that's why
> > `make` tends to update everything that is installed. Maybe Sam should
> > install additional packages in user scope, and then `make` won't try to
> > update them.
>
> I expect that the packages that update for Matthias on `make` are
> packages in "main-distribution"

Ah, no. I've helped Matthias when problems break his installation, I've
noticed that he installs packages not in "main-distribution" (e.g.,
"marketplace"), and I believe he really does want those updated.

I had that context in mind but didn't think to spell it out as I should
have.


> As an aside, the reason I don't install in user scope is that I switch
> between Racket implementations regularly, which would lead to
> out-of-date zo errors for all my user packages (instead, I get
> multiple copies of the packages).

Reply all
Reply to author
Forward
0 new messages