Should I stop sending packages to the catalog?

[Sage Gerard]

April 9th in the #general Slack channel taught me that there was no clean way to

release a breaking change in a package. I'm open to working on version pinning

support in raco pkg provided that a maintainer can walk me through some code.

In the meantime, as much as I appreciate the efforts made in the current system,

I'm considering stopping my contributions to the package catalog until further notice.

I'm open to submitting packages if I am confident in their longevity, but I don't want

to end up in the position I've been in for the last few weeks when planning a release.

That position being an inability to release an edition that is not in some way "aware"

of a prior edition. In my view, changing the package/collection name is an

example of that problem, not a solution to it.

I'm considering asking my users to specify different package sources in their info.rkt

files when dealing with my work. Before I commit to that decision, I wanted to tap into

those of you who have already been here. How have you handled breaking changes

for your projects? How have you communicated with your users to make sure they

were adequately prepared? Am I being too black-and-white about this?

~slg

[David Storrs]

+1 on this.  I'm having the same issue.

--
You received this message because you are subscribed to the Google Groups "Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/3tKIr3GrU9Jnl7V-5yzb52OL3mjt8XNe9F_Qv9HDKwy8xC4j9lQo2e5eGSle4ZFHAee_FiGVhr15lXoovUE6yqoARP-ZNi3kXXz8ETXdufg%3D%40sagegerard.com.

[Alex Harsanyi]

You could both send packages to the package catalog and instruct your users to use a different package source if they wish to use old versions.  I don't see these two options in conflict with each other.

As an application writer, I am on the other side of this problem, by depending on other packages. Having limited time to work on my project I want to upgrade package dependencies at my own pace.   The solution I found is to use git submodules and add each package as a submodule to my application.  I  construct a catalog of these packages, at an exact git commit, and simply run a raco pkg install, knowing that the exact version that I want will be there.  From time to time, I update these submodules and allow some time for fixing any problems that might result from these updates.  This has worked reasonably well so far, and I can reliably reconstruct older versions of the application knowing that all dependencies will be correct for that old version.

For my part, for any packages that my application depends on, their authors can make any breaking changes they want, as this will not immediately break my application.  At a later date I can review these changes, update my application and move to the new version.

The only downside is that installed packages are at "user scope" rather than "application scope" so once installed from the git submodules, that package becomes available to all applications on my computer.   So far this has not been a real issue, although I had to uninstall, switch catalogs and re-install a few times.

I personally regard pkgs.racket-lang.org as the source for the latest package versions, for users which just started using the package. Users who need precise versioning, can always setup their own package catalogs (i discovered that it is a really simple process)

Alex.

[Laurent]

Alex, that looks like an interesting workflow. Maybe worth a blog post? ;)

--
You received this message because you are subscribed to the Google Groups "Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/b3b54ccf-b4f1-425e-802a-110d5ac26591%40googlegroups.com.

[Jay McCarthy]

Hi Sage,

Racket has a very high standard of backwards compatibility where we
strive for programs to work continuously after many many years. In
most cases, we've been able to live up to the standard with the core
pieces of Racket. I can think of only a few times where we broke
compatibility and they were really traumatic and I think we learned a
lot from them about what compatibility really means.

The package system reflects this ethos. As you know, you can give a
dependency a lower-bound, but not an upper-bound, because we have
always been fairly loose with adding functionality (although it is
technical backwards incompatible), but not removing functionality. If
you need to break old programs, then you need to make a new logical
package, i.e. choose a new name, perhaps by adding a number to your
old name.

If you need to remove functionality or otherwise break old code, then
you are not really working on the same package anymore, but a new
product with similar functionality. In the history of Racket, you can
see this with the change from `(module _ mzscheme ...)` to `#lang
scheme` to `#lang racket` and in the change from `(require mred)` to
`(require scheme/gui)` to `(require racket/gui)`. These are NOT just
advertising changes because of new names; instead, the name was a
result of the need to add or remove functionality, while still
allowing all old programs to continue working.

This is simply a social standard though. There is nothing that
technically prevents you from breaking compatibility, except that your
users may be upset. You can post things on the package server that
follows any rules you want, including conflicting with any other
packages.

If you do so, and if your users need to pin themselves to particular
versions there are broadly two techniques. First, you can create your
own package catalog. Most users typically use two catalogs: the "big"
one on pkgs.racket-lang.org and another "small" one for your
particular release, where the core racket packages are pinned to
particular versions. When we were designing the package system, we
imagined that organizations would maintain their own catalogs of
audited versions of packages so they would not be surprised by any
automatic updates. Second, you can change your package source to pin
to a particular git revision/tag. This might be particularly useful if
a package relies on erroneous behavior of some package that was
otherwise fixed.

Jay

--
Jay McCarthy
Associate Professor @ CS @ UMass Lowell
http://jeapostrophe.github.io
Vincit qui se vincit.

On Wed, Apr 29, 2020 at 12:47 PM Sage Gerard <sa...@sagegerard.com> wrote:
>

> --
> You received this message because you are subscribed to the Google Groups "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/3tKIr3GrU9Jnl7V-5yzb52OL3mjt8XNe9F_Qv9HDKwy8xC4j9lQo2e5eGSle4ZFHAee_FiGVhr15lXoovUE6yqoARP-ZNi3kXXz8ETXdufg%3D%40sagegerard.com.

[Sage Gerard]

Hi Jay,

Thanks for the recap. I suspected that I had enough context,
so it helps to have confirmation from a core dev.

It seems I should leverage a patchwork of sources to support
different use cases.

~slg

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

[Jesse Alama]

On Thursday, April 30, 2020 at 2:57:45 PM UTC+2, Jay McCarthy wrote:

This is simply a social standard though. There is nothing that
technically prevents you from breaking compatibility, except that your
users may be upset. You can post things on the package server that
follows any rules you want, including conflicting with any other
packages.

I'd like to second this point. There's nothing stopping you from pushing whatever you want to your repo, and hence distributing whatever you want via the package server. I've pushed breaking changes to my packages before, and no one has complained, so I guess I didn't break any part of the interface that they were using. (Or I have no users of my stuff at all, which is certainly possible!)

I don't know how many packages mention, in their description, that they're experimental, explicitly warning me that the interface is unstable and likely to change. I use 'em anyway because they offer useful functionality. I don't recall being nailed by breaking changes, but perhaps I'm just getting lucky.

What exactly is the claim, anyway, about the package server not allowing breaking changes? Is it that if you do a breaking change to your package, then it's possible that other people's packages correspondingly break? If so, then I think that's not a very interesting claim. Does the claim at issue just amount to a restatement of the ethos that Jay mentioned about trying to ensure backwards compatibility for a long time?

(All this said, I'd like to learn more about setting up custom package catalogs, as Alex mentioned, to take matters even more into your own hands.)

[Hendrik Boom]

Is there a mechanism for, when you know you are making a breaking change
in a package, at least being warned about other packages that may break
as a result?

And is there a mechanism for testing those other packages before
committing your breaking package to the public repository?

-- hendrik

>
> --
> You received this message because you are subscribed to the Google Groups "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/04ebf61c-54b3-4f58-96aa-f5bdc5f2b457%40googlegroups.com.

[Laurent]

For your second point, you can never really know what other work depends on your package. You could display a message in the new package or in the readme for example, but that's likely not going to work well for various reasons. (but see my last comment)

However, what Jay is saying I believe (correct me if I'm wrong), is that you can just create a tag for the previous version, modify your current package 'my-package' on the pkg server to refer to that tag, and create a new package 'my-package2' on the pkg server pointing to master. The pkg catalog is going to be cluttered with all backward-incompatible version numbers but a) Jay doesn't seem to be worried about that b) it's currently not the case (meaning, people don't do that currently, but maybe they should?).

Old users won't see the breaking change, and new users will see and use the new version on the catalog (or clone directly from github). What's missing again is a way to tell old users that a new package is available with new features, but I think Jay was saying some time ago that this should be the purpose of mailing lists and other media.

To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/20200501132344.4nwjgebdlyttwkzu%40topoi.pooq.com.

[Bogdan Popa]

Alex Harsanyi writes:

> As an application writer, I am on the other side of this problem, by
> depending on other packages. Having limited time to work on my project I
> want to upgrade package dependencies at my own pace.

I'm in a similar position since I operate a few Racket applications.

I'd been mulling over the idea of a snapshotting service for the main
package catalog to help automate some of this over the past couple of
months, but this thread finally pushed me to get started on building it.

More details here: https://github.com/bogdanp/racksnaps

It's not quite ready for general use yet (the first snapshot is being
built as we speak), but I should be able to iron out the kinks over the
next couple of weeks.

I'm curious what folks think about this approach.

[Sam Tobin-Hochstadt]

This is very cool! One question -- for "main-distribution" packages,
are you snapshotting the most-recent release catalog? Or
pkgs.racket-lang.org? The latter is simpler, but the former is
probably needed to make the examples in the README work. Otherwise,
if, say, "typed-racket-lib" on May 1 depended on a newer-than-7.6
version of "base", using the snapshot first (or only the snapshot)
would get you a version of "typed-racket-lib" that didn't work on 7.6.
I think that if you snapshot pkgs.racket-lang.org, then just adding
the 7.6 catalog before the snapshot catalog would work the way you
want, but I haven't fully thought through everything.

Sam

> --
> You received this message because you are subscribed to the Google Groups "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/m2k11vmw0d.fsf%40192.168.0.142.

[Bogdan Popa]

Sam Tobin-Hochstadt writes:

> This is very cool! One question -- for "main-distribution" packages,
> are you snapshotting the most-recent release catalog? Or
> pkgs.racket-lang.org? The latter is simpler, but the former is
> probably needed to make the examples in the README work. Otherwise,
> if, say, "typed-racket-lib" on May 1 depended on a newer-than-7.6
> version of "base", using the snapshot first (or only the snapshot)
> would get you a version of "typed-racket-lib" that didn't work on 7.6.

At the moment, I'm snapshotting pkgs.racket-lang.org but not updating
packages from the main distribution. That is, I start with the main
distribution of regular Racket 7.6 and add packages on top, but I don't
change any existing ones.

I think I can change the first example in the README to just

raco pkg config --set catalogs \
https://racksnaps.defn.io/snapshots/2020/05/01/catalog/ \
https://pkgs.racket-lang.org \
https://planet-compats.racket-lang.org

without changing the behavior.

[Sage Gerard]

As long as I don't have to keep reorganizing my code to accommodate the tooling, then it's a night and day improvement. Would you mind terribly if I worked with you? I was mulling over this myself in the dev list, but I am happy to aid any existing effort.



-------- Original Message --------

--

[Alex Harsanyi]

I think that there are two issues here:

First there's backwards compatibility -- it is a good idea to keep your packages backwards compatible, although sometimes it can be difficult, especially with packages which were released too early, before their interface stabilized.

Than there are breaking changes.  This does include the backwards incompatible changes, but also includes defects.  Defects are unintentional, usually unknown to the author at the time of publishing the new package version, yet they can break the application. 

As an application author, when my application stops working after updating packages, it is little consolation that the API of a package remained backwards compatible.

Perhaps I was just unlucky, but I encountered show-stopper defects in the first two external packages I tried to use in my application. I had a local fix, but had to wait for the author several months to fix the problem (this is understandable, the author was busy and these packages were not his top priority).   The mechanism I came up with was driven by the following needs:

    (1) sit out a package version because it has a defect

    (2) while sitting out a package version, I want to be able to update other packages that are not affected by the defect

    (3) use an alternate source for a package which contains a fix, until an official fix is released

    (4) delay updating packages, until I have time to verify that the update does not break the application and/or address problems

    (5) be able to reconstruct old software configurations including dependencies

Number (5) is very important to me: I have some development branches which are active for several months, and when I go bad to an older branch, to continue work, the last thing I want is to deal with a package upgrade -- this perhaps reflects personal circumstances: I quite often have 1 hour to try to progress a feature on an old branch, and if I spend 15 - 20 minutes merging master, updating packages and resolving conflicts, that's 25 - 30% of available time gone.

 

(All this said, I'd like to learn more about setting up custom package catalogs, as Alex mentioned, to take matters even more into your own hands.)

There is not much to it:  git submodules are widely documented, so I'll leave those out, but here is the script that sets up the package catalog

https://github.com/alex-hhh/ActivityLog2/blob/master/etc/scripts/setup-catalog.sh

And here is the snipped from the build file which sets things up and installs packages (note that this install uses "isolation mode" to catch dependencies which are not tracked as submodules, don't use the -i switch on your development machine):

      

bash ./etc/scripts/setup-catalog.sh -i pkgs/

 raco pkg install --auto tzinfo tzgeolookup data-frame plot-container

I also have added code to my application to verify that the packages are installed before building or compiling it and it produdces more user friendly error messages. (e.g. it tells the user that they forgot to install the "tzinfo" package, rather than failing with a "module not found" error).  You can find that here: https://github.com/alex-hhh/ActivityLog2/blob/777847f2cdb18ff09596848efc40a332e65da017/rkt/utilities.rkt#L446   Alex.

 

[Bogdan Popa]

Sage Gerard writes:

> As long as I don't have to keep reorganizing my code to accommodate
> the tooling, then it's a night and day improvement. Would you mind
> terribly if I worked with you? I was mulling over this myself in the
> dev list, but I am happy to aid any existing effort.

I'd appreciate it! Let's sync up on Slack.

[Alex Harsanyi]

On Thursday, April 30, 2020 at 3:59:49 PM UTC+8, Laurent wrote:

Alex, that looks like an interesting workflow. Maybe worth a blog post? ;)

Well, it took longer than I anticipated, but here it is: https://alex-hhh.github.io/2020/05/dependency-management-in-racket-applications.html

I understand that everyone has a different workflow and my workflow will not be directly applicable to them, but hopefully a few ideas can be re-used.  I also tried to explain what I would like from a "dependency management" solution.

Alex.

 

To unsubscribe from this group and stop receiving emails from it, send an email to racket...@googlegroups.com.

[Brian Adkins]

On Thursday, April 30, 2020 at 8:57:45 AM UTC-4, Jay McCarthy wrote:

If you do so, and if your users need to pin themselves to particular
versions there are broadly two techniques. First, you can create your
own package catalog. Most users typically use two catalogs: the "big"
one on pkgs.racket-lang.org and another "small" one for your
particular release, where the core racket packages are pinned to
particular versions. When we were designing the package system, we
imagined that organizations would maintain their own catalogs of
audited versions of packages so they would not be surprised by any
automatic updates. Second, you can change your package source to pin
to a particular git revision/tag. This might be particularly useful if
a package relies on erroneous behavior of some package that was
otherwise fixed.

I posted a message in Slack about this also, but in case the mailing list is preferred for whoever has the info, I'll post here also.

I'm curious about this approach. The scenario that's important to me is being able to control the versions of packages that my app uses. If a package is updated, and introduces a bug, I don't want that to cause failures in my production app i.e. I want the same set of packages (at specific versions) that I'm developing and testing with on my laptop to be identical to the ones I will eventually install on my servers at a later point in time.

Are people using this approach, described by Jay, of hosting their own catalogs, to accomplish this? If so, how do you get setup? In other words, how do you populate the catalog with the specific versions of packages you want? 

[Neil Van Dyke]

For an important production system, you probably want the source of any
third-party packages on which you depend to be in Git (or another SCM
system) that you control.

You might also want to audit those packages yourself, as well as audit
any new version changes to them, before you push to production.

After you do those things in SCM, depending how you do it, you *might*
find it's more convenient to simply load the third-party code you need
using the module system `require` only, without an additional package
system.

[Brian Adkins]

While I see some benefits of this approach, I just looked at a typical Rails project of mine, and it has over 160 packages with some packages depending on different versions of other packages, etc., so I think managing all of this myself in git might be overly burdensome. Granted, my current Racket apps have far fewer dependencies, but I expect that may increase over time. 

[Hendrik Boom]

If you want to avoid problems with software you use changing
unpredictably, you will have to have use your own copy that does not
change at all.

I don't see another alternative if your upstream source might change at
any time.

And you will likely have to vet any chaanges that appear upstream in
case they actually do fix security-related issues that might already
be affect you. It will be up to you to determine the risks of leaving
your local copy unchanged.

-- hendrik

>
> --
> You received this message because you are subscribed to the Google Groups "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/32b8f70a-4033-43db-aca3-b30b9023f3a3o%40googlegroups.com.

[Brian Adkins]

I'm hoping the main scenario I'm concerned about (a bug is introduced in an update of a package) is rare, although that's exactly what motivated Alex to create his system. If it is a rare scenario for me, then I'd like to do the following:

1) Develop & test locally while updating packages as needed

2) Prior to releasing to production, so *something* that effectively snapshots my environment

3) Use the snapshot to deploy to production

Although I have some ideas about the "something" task, I still have to finalize the procedure and try it out. One option is to simply use a racksnap snapshot. What I'd like to do is to create a personal catalog that represents the packages I currently have installed, but it doesn't appear there's a super easy `raco pkg` command to do that - I still need to research a bit. Either there is a command, or it seems like a relatively straightforward thing to add.

[Hendrik Boom]

On Fri, Jun 19, 2020 at 01:09:08PM -0700, Brian Adkins wrote:
>
> I'm hoping the main scenario I'm concerned about (a bug is introduced in an
> update of a package) is rare, although that's exactly what motivated Alex
> to create his system. If it is a rare scenario for me, then I'd like to do
> the following:
>
> 1) Develop & test locally while updating packages as needed
> 2) Prior to releasing to production, so *something* that effectively
> snapshots my environment
> 3) Use the snapshot to deploy to production
>
> Although I have some ideas about the "something" task, I still have to
> finalize the procedure and try it out. One option is to simply use a
> racksnap snapshot. What I'd like to do is to create a personal catalog that
> represents the packages I currently have installed, but it doesn't appear
> there's a super easy `raco pkg` command to do that - I still need to
> research a bit. Either there is a command, or it seems like a relatively
> straightforward thing to add.

Making a complete copy of everything stored under ~/.racket might make
enough of a racket system snapshot on a GNU/Linux system. I have no
idea what to do on Windows, though.

-- hendrik

[Simon Schlee]

What I'd like to do is to create a personal catalog that represents the packages I currently have installed, but it doesn't appear there's a super easy `raco pkg` command to do that - I still need to research a bit.

raco pkg archive --help
Create catalog from installed packages

Usage:
raco pkg archive [ <option> ... ] <dest-dir> <pkg> [<pkgs>] ...
 where <option> is one of
  --include-deps : Include dependencies of specified packages
* --exclude <pkg> : Exclude <pkg> from new catalog
  --relative : Make source paths relative when possible
  --help, -h : Show this help
  -- : Do not treat any remaining argument as a switch (at this level)
 * Asterisks indicate options allowed multiple times.
 Multiple single-letter switches can be combined after one `-'; for
  example: `-h-' is the same as `-h --'
 

[Brian Adkins]

Yup. And then either list all the packages individually, or create an uber package that lists all the packages individually and archive that. Not very convenient, so I may look into adding a flag that creates a catalog from *all* of the locally installed packages.

[Simon Schlee]

Declaring the dependencies explicitly is good in a lot of cases, but I guess there are cases where it is unnecessary.

If you do decide to create the uber package you can create the info.rkt, `raco pkg install` the package and then use `raco setup --fix-pkg-deps --unused-pkg-deps <collection-name>` to let setup create the list of package dependencies.