Testing for Packages

[Robert Postill]

Hi, 

Inspired by the reply to https://stackoverflow.com/questions/62014612/how-to-test-a-racket-package-self-sufficiently, I decided to ask the question here.  So a little background to the discussion.  I've been looking at a racket package, and the package has a couple of collections.  It's been dormant for a few years, and so when I ran the tests they didn't work (it's a driver, and the database it works with had moved on in the intervening time).  So I started thinking about the tests.  One thing that troubled me was that the tests seemed to be dependant on the package already being installed. Ryan Culpepper's comment on the stack overflow question (https://stackoverflow.com/a/62027185/11219) suggests that the tests should be run once the package is under raco pkg's management. Up until that point, I had been considering a PR rewiring the require statements to be significantly more relative.

In the generally excellent docs for racket, I haven't found advice relating to the right way to structure the tests for a package. Nor an idiomatic approach to dealing with package installation during package development. Here are my sources for reference:

https://blog.racket-lang.org/2017/10/tutorial-creating-a-package.html

https://beautifulracket.com/jsonic-3/the-package-server.html

https://greghendershott.com/2013/12/racket-package-management.html

http://www.greghendershott.com/2017/04/racket-makefiles.html

Now I think I should explain why I'm thinking that way.  FYI I've been a part of the ruby community, and so my thinking has a ruby-ish colour.  Which makes me think I'm maybe off in my community norms for racket. I've always considered the point at which a package gets transferred into the domain of raco pkg's management to be a pivotal moment. It has this implied social contract that if I'm mixing with code into a system area managed by the language, the package needs to be trustworthy. Testing *after* installing seems a bit like I'm putting the cart before the horse? It feels like saying here's my code, put it in a system-wide place, I hope it works.

So I'm interested in opinions and advice as to how to think correctly about testing packages and also how other people approach creating a well-behaved package.

Regards

Robert

[Alexis King]

> On May 26, 2020, at 21:36, Robert Postill <robert....@gmail.com> wrote:
>
> One thing that troubled me was that the tests seemed to be dependant on the package already being installed.

I think this is very common within the Racket ecosystem. I would guess that very few packages are consciously designed to be “relocatable” in the way you describe.

> I've always considered the point at which a package gets transferred into the domain of raco pkg's management to be a pivotal moment. It has this implied social contract that if I'm mixing with code into a system area managed by the language, the package needs to be trustworthy. Testing *after* installing seems a bit like I'm putting the cart before the horse?

I cannot speak to the cultural norms of the Ruby community, as I am not terribly familiar with them, but I will admit that I find this mentality somewhat curious. In Racket, “installing” a package doesn’t affect very much beyond module path resolution. Running a package’s tests can do all the same things—install a keylogger, steal your private key, encrypt your hard drive—whether it is installed or not. Installation is also not very permanent; it’s very easy to undo or change.

So from my perspective, going out of your way to download package code and run it without officially installing it via `raco pkg` seems like a mostly pointless exercise. You’re basically doing all the same things—downloading it, unpacking it to a particular location on your computer, and giving it all the privileges of the user you run it as—just manually, rather than letting raco do the work for you. It is entirely possible that there is some additional distinction or utility I am not seeing, but my first instinct is to recommend that you just install the package. You can always uninstall it.

Alexis

[Alex Harsanyi]

I am personally confused about what you think the problem is, so I just want to make sure that you know that you can develop and test a new package, or you can modify an existing package and test the modifications, without publishing the package.

Basically in Racket you can:

* create a new package using "raco pkg new", or clone an existing one

* install the package from the local folder, using "raco pkg install ./my-package"

* make changes and run the tests -- you can keep making changes on the installed package without having to re-install it

* once the tests pass, you can commit the changes and publish them

"Under raco pkg management" does not mean that the package is publicly available on the racket package server.  You can keep the package private, and still be able to install it and run the tests.

Alex.

[George Neuner]

On 5/26/2020 10:36 PM, Robert Postill wrote:

Inspired by the reply to https://stackoverflow.com/questions/62014612/how-to-test-a-racket-package-self-sufficiently, I decided to ask the question here.  So a little background to the discussion.  I've been looking at a racket package, and the package has a couple of collections.  It's been dormant for a few years, and so when I ran the tests they didn't work (it's a driver, and the database it works with had moved on in the intervening time).  So I started thinking about the tests.  One thing that troubled me was that the tests seemed to be dependant on the package already being installed. Ryan Culpepper's comment on the stack overflow question (https://stackoverflow.com/a/62027185/11219) suggests that the tests should be run once the package is under raco pkg's management.

The problem is that any particular package may require a specific runtime environment:  e.g., many packages provide Racket interfaces to 3rd party libraries.  And those libraries have their own sets of dependencies.  And many libraries have compatibility problems with others, so having many libraries available in a "generic" test environment is not possible. 

But even if the package host did maintain custom test environments (VM or container or ...) for every contributed package - that STILL does not guarantee that any given package will work on YOUR system, with YOUR particular set of libraries.

You say the package that inspired this question used a database.  Which database?  What version?  There are dozens of DBMS available - many are NOT $$$ free to own, and all of them have multiple versions having various compatibility problems (i.e. what works with v8.2 may not work with v10.4 - or the reverse).  Many server class DBMS will not peacefully co-exist with another on the same system, so then you're talking about many virtual machines or containers (if the DBMS in question even works in a container - some don't). 

In the generally excellent docs for racket, I haven't found advice relating to the right way to structure the tests for a package.

Right.  Because many packages interface to 3rd party software, and so a lot of - so called - advice would, of necessity, be generic to the point of being useless.

Now I think I should explain why I'm thinking that way.  FYI I've been a part of the ruby community, and so my thinking has a ruby-ish colour.  Which makes me think I'm maybe off in my community norms for racket. I've always considered the point at which a package gets transferred into the domain of raco pkg's management to be a pivotal moment. It has this implied social contract that if I'm mixing with code into a system area managed by the language, the package needs to be trustworthy.

If the Ruby maintainers told me that they were able to test every contributed package in that package's expected runtime environment, I simply would not believe them.

I wouldn't believe it if Google, or Apple, or Oracle, or Microsoft, or Redhat, or Ubuntu said it either.  Nobody can do it - the expense is prohibitive.

Testing *after* installing seems a bit like I'm putting the cart before the horse? It feels like saying here's my code, put it in a system-wide place, I hope it works.

I understand both the motivation and the frustration.  For a long time I developed embedded software under situations where I controlled everything: the hardware, the OS (if there was one), my application, the libraries it depended on, etc.  I knew exactly the conditions my software was expected to operate under, and and the complete set of input it could possibly receive.

But 99+% of all software has to run under an uncertain environment that, at best, is only partly know to its developer, depending (at least transitively) on a lot of 3rd party software that the developer often is completely unaware of.


YMMV,
George