New version of file-watchers

63 views
Skip to first unread message

Sage Gerard

unread,
Aug 28, 2019, 6:30:04 PM8/28/19
to users\@racket-lang.org
Version 0.2 of the file monitoring package file-watchers is about to drop. Here's a PR for public review. There are no contributors, so I plan to merge in 48 hours if there is no feedback. Otherwise I'll play it by ear. https://github.com/zyrolasting/file-watchers/pull/4

The new release upgrades the CLI and the public API. Highlights:
  • The CLI provides cleaner feedback on what ends the process.
  • All file monitoring approaches can now target individual files as well as directories. Earlier only directories could be specified, which make it hard to ignore files.
  • (watch-directories) is deprecated in favor of (watch), which supports files and directories. The former will be removed after Jan 1st 2020.
~slg


Stephen De Gabrielle

unread,
Aug 31, 2019, 3:07:49 AM8/31/19
to Sage Gerard, users\@racket-lang.org
Awesome -thank you for releasing this.

s.


--
You received this message because you are subscribed to the Google Groups "Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/GDZKfX5RNbeKTSVXJg7DaBOp9MqBqz84s_-fuZ8reIHqUr1ewKclTSq7VDq8kYq17Bc466KH85pRVh6QD_r6T1C8MDRdWqFfYS_-Jfby9wQ%3D%40sagegerard.com.

Neil Van Dyke

unread,
Aug 31, 2019, 7:00:49 AM8/31/19
to Sage Gerard, us...@racket-lang.org
First, to Sage, thank you for going to the work to open source
third-party packages like this.

Here are a few suggestions that came to mind from this post, which I'd
like to mention to Racket third-party package developers in general.
(Only suggestions, not presuming to tell anyone what to do; just
avoiding real work this early Saturday morning. :)  Also, as I was
writing this, I realized one of the points is an opportunity to give
some context on recent governance concerns.

Note that some of these suggestions are biased to a development approach
that encourages developing all software in such a way that there's often
low friction to project/organization-internal component reuse, or even
to selectively open-sourcing individual modules.

* Normally, you'll be developing your own large systems, and a given
open source package will just be a small part of it, and you might have
many such open source packages.  You probably don't want to be slowing
yourself down with offering people previews of new versions, lead times
for input, etc. -- when you're ready to push a new version, just do it. 
There are other mechanisms for not breaking users of your package.  Of
course you can ask for input if you're contemplating big or breaking
changes, and not sure what to do.

* Try not to make identifiers be simple generic terms, like `watch`,
that might better be used for key language features, or in users' own
programs.  (Note that, historically in Racket, there was an idea that
modules might provide such simple generic identifiers, and then modules
using them would add a prefix or otherwise transform them. But that had
a couple problems: it was more work for the module user in practice, and
it also meant that reading unfamiliar code or looking up unfamiliar
identifiers in documentation/Google search was harder.  There will
always be a lot of name collisions in Racket documentation search of
core Racket and whatever third-party modules are installed, but
non-generic names improves that in a worse-is-better kind of way.)

* Of course, you can name your identifiers however you want, and you
might want to make them fairly descriptive, in the scope of all Racket
packages.  One convention that sometimes (not always) seems good is to
include the package name in each exported identifier, which helps search
and recognizing which package an identifier comes from when reading
code, and also reduces name collisions between packages.  For example, I
think every identifier from the `roomba` package contains `roomba` in
its name (and if you wanted shorter names for this package, to make it
more like, say, Logo turtle graphics, that might be a `#lang`),
"https://www.neilvandyke.org/racket/roomba/  That convention might not
be appropriate for `file-watchers`, but I just wanted to mention it,
because it seems to often help.

* Try not to remove deprecated interfaces from within a package, unless
keeping them is a significant burden for you.  You can move the
documentation for them to some dark corner of the manual (like the end
of the manual, or end of a section), and use the `deprecated` Scribble
procedure on them.  But if it's practical to keep them in, that reduces
breaking users of your modules.  Think of a little deprecated API as
like scars of a battle-tested and enduring package.

* When you do want to break the interfaces, there's currently no great
way.  Racket's original PLaneT package system had a very simple yet very
nice SemVer-like versioning system that some third-party package authors
used,[1] and multiple versions of the same package could be installed at
once.  That was particularly helpful for low-encumbrance third-party
packages.  Users of the package could keep using the old version until
they were ready to make changes to use the new one, and the package
developer (usually altruistically releasing some part of their system
that was their real job) wasn't unduly burdened.  And being able to have
multiple versions of a package installed at once could often (though not
always) let you move incrementally, of packages you control, and from
the graph of interdependent third-party packages you use.  With the new
package system, that version-related functionality was lost, and it was
emphasized as policy that new package versions should always be
backwards compatible, and if you wanted to break an interface, you
should make a new package (with a new name).  Losing that various
version-related functionality made sense for core Racket's policy change
to support backward-compatibility, but I think it didn't make sense for
the breadth of third-party developers for whom we'd like to encourage
low-friction release and updating of their packages.[2]


[1] See
"http://planet.racket-lang.org/display.ss?package=javascript.plt&owner=dherman",
which was up non-backward-compatible version 9.  Dave Herman was able to
keep doing his work, but also keep releasing new versions. People who
used that could depend on it and update at their convenience.  You'll
also see use of that in many of his other packages at
"http://planet.racket-lang.org/display.ss?owner=dherman", and also use
of non-backward-compatible versioning by other people at
"http://planet.racket-lang.org/".  I also used this feature for various
big and small packages,
"http://planet.racket-lang.org/display.ss?owner=neil".

[2] This was a great learning moment for Racket open source, and
relevant to the disruption with governance and Racket2.  How things
happened was very costly for some third-party developers and for morale,
and we lost at least one of our prolific contributors and a nice book
over it.  If anyone is confused why some Racketeers recently see to feel
strongly about being clear about exactly what will drive Racket
decisions and how that will be made (top-level requirements, process,
etc.) -- one reason is the history of Racket itself (this and other
events), besides all the things we've learned from countless other open
source and industry projects.  It's easy to see fairly new Racketeers
understandably making unwarranted assumptions about the goals and where
things will go.  Being unambiguously clear about what drives Racket
decisions, and being able to trace everything back to those goals
(through multiple decisions/requirements), seems very healthy for both
new and remaining long-time Racketeers.  It will also help focus
discussion about Racket2, once things get past brainstorming (e.g., save
people a lot of time working on details that would trace back to
mutually-exclusive higher-level requirements).

Sage Gerard

unread,
Aug 31, 2019, 10:38:55 AM8/31/19
to Neil Van Dyke, us...@racket-lang.org
Hi Neil, and thanks for the helpful information and historical context.

I do have some follow up questions if that's alright, because I want to make the most out of my time here and it seems like you can clarify that.

> You probably don't want to be slowing yourself down with offering people previews of new versions, lead times for input, etc.

I'm unknown in open source so creating as many opportunities as possible for feedback is valuable enough to be worth the effort. On Aug 25th I asked on this list if regular updates would come off as bad etiquette or unwanted solicitation, and the answer I got seemed to suggest that it would be alright. By the subject line of the email it seems that the thread was cannibalized. It seems I misinterpreted the earlier answer, and for that I apologize.

> Identifier naming

I felt the pain over the identifier xexpr->string. I personally didn't particularly mind using prefix-in to address it, but I chalked it up to how namespaces were designed and went with it.

But I'm wondering: How much do volunteers have to change the way they work due to technical limitations out of their control? I understand that to the extent that these are not my servers and other people are impacted by what I put out. Where I get lost is that--from what I can tell--this is a community of people who wish to work in ways that other designers might not approve and I thought the ecosystem wouldn't raise a flag over a provided name.

> When you do want to break the interfaces, there's currently no great way.

I'm already seeing a pattern here: Do technical limitations define governance of volunteers?

To put it another way, do you allow design limitations to draw a hard line between regulating contributions and allowing contributors to have their own way of doing things?

Re: deprecation, I strongly believe that less code is better, and I hope that I can draw compromises with users on doing things like remove redundant functionality on my own. If several months notice for removal is another thing that raises a flag, then I'll refresh myself on the style guide and of course take your suggestions to heart to improve the experience for everyone.

But... I do also want my way of working (for free, no less) to be welcome to the greatest extent possible. What can I do to give myself and other volunteers that space?

~slg

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Saturday, August 31, 2019 7:00 AM, Neil Van Dyke <ne...@neilvandyke.org> wrote:

> First, to Sage, thank you for going to the work to open source
> third-party packages like this.
>
> Here are a few suggestions that came to mind from this post, which I'd
> like to mention to Racket third-party package developers in general.
> (Only suggestions, not presuming to tell anyone what to do; just
> avoiding real work this early Saturday morning. :)  Also, as I was
> writing this, I realized one of the points is an opportunity to give
> some context on recent governance concerns.
>
> Note that some of these suggestions are biased to a development approach
> that encourages developing all software in such a way that there's often
> low friction to project/organization-internal component reuse, or even
> to selectively open-sourcing individual modules.
>
> - Normally, you'll be developing your own large systems, and a given
> open source package will just be a small part of it, and you might have
> many such open source packages.  You probably don't want to be slowing
> yourself down with offering people previews of new versions, lead times
> for input, etc. -- when you're ready to push a new version, just do it. 
> There are other mechanisms for not breaking users of your package.  Of
> course you can ask for input if you're contemplating big or breaking
> changes, and not sure what to do.
>
> - Try not to make identifiers be simple generic terms, like `watch`,
> that might better be used for key language features, or in users' own
> programs.  (Note that, historically in Racket, there was an idea that
> modules might provide such simple generic identifiers, and then modules
> using them would add a prefix or otherwise transform them. But that had
> a couple problems: it was more work for the module user in practice, and
> it also meant that reading unfamiliar code or looking up unfamiliar
> identifiers in documentation/Google search was harder.  There will
> always be a lot of name collisions in Racket documentation search of
> core Racket and whatever third-party modules are installed, but
> non-generic names improves that in a worse-is-better kind of way.)
>
> - Of course, you can name your identifiers however you want, and you
> might want to make them fairly descriptive, in the scope of all Racket
> packages.  One convention that sometimes (not always) seems good is to
> include the package name in each exported identifier, which helps search
> and recognizing which package an identifier comes from when reading
> code, and also reduces name collisions between packages.  For example, I
> think every identifier from the `roomba` package contains `roomba` in
> its name (and if you wanted shorter names for this package, to make it
> more like, say, Logo turtle graphics, that might be a `#lang`),
> "https://www.neilvandyke.org/racket/roomba/  That convention might not
> be appropriate for `file-watchers`, but I just wanted to mention it,
> because it seems to often help.
>
> - Try not to remove deprecated interfaces from within a package, unless
> keeping them is a significant burden for you.  You can move the
> documentation for them to some dark corner of the manual (like the end
> of the manual, or end of a section), and use the `deprecated` Scribble
> procedure on them.  But if it's practical to keep them in, that reduces
> breaking users of your modules.  Think of a little deprecated API as
> like scars of a battle-tested and enduring package.
>
> - When you do want to break the interfaces, there's currently no great

Sage Gerard

unread,
Aug 31, 2019, 10:45:11 AM8/31/19
to Sage Gerard, Neil Van Dyke, us...@racket-lang.org
I looked back and found my mistake: I only asked about blog posts and videos. I'll refrain from version update posts here from now on.


~slg

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> --
>
> You received this message because you are subscribed to the Google Groups "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/JK5AJveYoI3I7H_pRK0IPbhsFfPmRzaaoZW7HF4eOLEHgmAnULv5Heu-fJG4vGsH3P2Ntu698BqIglKnslXXykQZU_v-NTzet7AsF_53uOg%3D%40sagegerard.com.


Neil Van Dyke

unread,
Aug 31, 2019, 12:32:04 PM8/31/19
to Sage Gerard, us...@racket-lang.org
Sage Gerard wrote on 8/31/19 10:38 AM:
>> You probably don't want to be slowing yourself down with offering people previews of new versions, lead times for input, etc.
> I'm unknown in open source so

I think all of us in Racket land are pretty much unknown in open source
(outside of Racket/Scheme). :)

> creating as many opportunities as possible for feedback is valuable enough to be worth the effort.

OK, if you want the input, or are willing to do that extra work, that's
great.  I only meant it as I said it: I wanted people considering being
third-party package developers to know they weren't obligated to take on
that extra burden if they don't want to.

> Re: deprecation, I strongly believe that less code is better, and I hope that I can draw compromises with users on doing things like remove redundant functionality on my own.

That sounds reasonable to me, if that's how you want to do it for your
packages.

If a third-party package developer expects to possibly break
backward-compatibility in the future, it might make sense for them to
indicate that near the top of their package documentation.  That makes
the social contract more clear.

(I don't think there was much buy-in on that backwards-compatibility
policy for third-party packages, which was handed down to replace the
lost PLaneT version support.  It was core-centric, and burdensome to
third-party developers who are doing rapid/agile
research/move-fast-and-break-things, and/or open-sourcing parts of a
system on which they do things like global refactoring, and want to be
able to make backward-incompatible changes to their code.)

> the style guide

That style guide applies only to the code of core Racket itself, not to
third-party packages.  You can look at it for ideas, or adopt it as a
bible, if you want, but do your own code however you like.


I think a general theme in these comments is to encourage important and
sensible things for package developers (e.g., create package metadata
and catalog entry, do a versioned release of some kind, consider
including documentation and tests, consider naming things to work well
with doc search and code readability), but not burdens that people will
often either ignore or find too discouraging to do anything at all
(e.g., you don't have to format your code a certain way, you don't have
to preserve backward compatibility with each version, you don't have to
Scribble).

Hendrik Boom

unread,
Aug 31, 2019, 1:04:20 PM8/31/19
to Sage Gerard, us...@racket-lang.org
On Sat, Aug 31, 2019 at 02:45:02PM +0000, Sage Gerard wrote:
> I looked back and found my mistake: I only asked about blog posts and videos. I'll refrain from version update posts here from now on.

Whatever you originally asked for, I think this ensuing discussion about
third-party packages is valuable.

-- hendrik

Sage Gerard

unread,
Aug 31, 2019, 7:27:39 PM8/31/19
to Hendrik Boom, us...@racket-lang.org
Thanks for the clarification, Neil! Your feedback does help a great deal. I'll also be happy to share updates with a respectful frequency now that I see the other replies. :)


~slg

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> -------------------------------------------------------------------------------------------------------------------------
>
> You received this message because you are subscribed to the Google Groups "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/20190831170409.w64sibufwjz5555y%40topoi.pooq.com.


Simon Schlee

unread,
Sep 1, 2019, 3:28:16 PM9/1/19
to Racket Users
Try not to make identifiers be simple generic terms, like `watch`

I strongly disagree with this statement.
Rackets require possibilities are highly flexible so if you want more specific names use prefix-in and prefix the module name or whatever you like.
The way racket does imports is one of the things I highly value because you can use modules to aggregate identifiers from different modules/packages,
resolve name conflicts by renaming / or not exporting the 2 identifiers that clash.
This creates a lot of freedom to create specific sets of functionality exported by one simple module.
This way I can create my custom set of things which are most needed in the particular codebase.
Instead of having to write 40 lines of requires like you have to in other languages.
Adding prefixes to imports is easy, having to create a prefix-remove that can be used in require to cut away prematurely added prefixes would annoy me.

Finding things in the documentation is a whole other technical issue.
If you find the documentation search features lacking for your search needs, we should figure out how the search could be improved, instead of making the code unnecessary verbose.
You already could use "watch M:<part-of-module-path>" to only find identifiers exported by modules that use that module path.

Neil Van Dyke

unread,
Sep 1, 2019, 4:55:20 PM9/1/19
to Simon Schlee, Racket Users
Simon Schlee wrote on 9/1/19 3:28 PM:
>
> Try not to make identifiers be simple generic terms, like `watch`
>
>
> I strongly disagree with this statement.

I think what you say is valid, and I agree in some situations.

However, we're talking about reusable third-party packages, which are
used pretty much the same as much of the "standard library" of Racket.

When I look at the Racket Reference, the standard library tends to use
descriptive names, like `file-or-directory-permissions` rather than
`permissions`.

A third-party package analog might be `watch-file-or-directory` (or
whatever the rough purpose is) rather than `watch`.

> The way racket does imports is one of the things I highly value
> because you can use modules to aggregate identifiers from different
> modules/packages,
> resolve name conflicts by renaming / or not exporting the 2
> identifiers that clash.
> This creates a lot of freedom to create specific sets of functionality
> exported by one simple module.
> This way I can create my custom set of things which are most needed in
> the particular codebase.
> Instead of having to write 40 lines of requires like you have to in
> other languages.

True, that kind of module coupling often happens, and a module to
aggregate a class of exports used by a codebase can be very convenient.

That aggregating module might also, as you said, be an opportunity to
fix the problems created by having all those modules export terse
generic names rather than more descriptive ones.

BTW, one drawback to the exports-aggregating module for a codebase comes
when any of the aggregated modules changes (or the exports change,
depending on how simple the compiler's sense of dependencies and
changes).  In a large system, for an aggregating module used widely,
this can cause needing to recompile much of the codebase (which can be
costly, in the size of system in which I think one would consider making
an aggregating module like that).  Imagine the perhaps not-unusual
situation of changing a module actually only used by 10% or 1% of the
modules that import the aggregating module.  Coincidentally, the largest
Racket system I worked on had a couple such aggregating modules, which
initially made sense, until they were covering way too much stuff that
got changed way too frequently, and it made the code base way more
cumbersome to work with than just having the appropriate `require`s
would be. (Auto-updating the `require`s would've been nice, though.) 
One of them actually made a lot of sense initially -- it was a logical
whole that was unlikely to change -- until that thing got more things
added to it, and until we needed to actively work on that thing while
testing it integrated with the system (unit tests aren't always
sufficient, or aren't always the most expedient).

> Finding things in the documentation is a whole other technical issue.

I agree that could be improved.  One bit of obvious low-hanging fruit
(with IDE integration) is having the context of your current #lang and
requires as factors in search hit rankings.  (I'm not criticizing the
current documentation search; it works surprisingly well for offline
search, and with not that much resources, and was very clever in how it
was implemented.  I think it might've been by Eli Barzilay.)

In the interim, a boost comes from descriptive names (which also happens
to be idiomatic Racket, and idiomatic Racket).  That boost might be
worse-is-better, but it works. https://en.wikipedia.org/wiki/Worse_is_better

Also, I don't think Web searches, such as of forum posts/messages,
articles, code repos, etc., are sensitive to ad hoc import renaming.  So
you might get better global search hits for "racket
file-or-directory-permissions" than "racket file permissions".

*

Racket2 aside, since you might be looking for terseness in some
regard... One of the higher-level (but not top-level) requirements for
Racket2 might concern terseness.  Not just in naming, but in other
things as well (e.g., does a mutating or definition form also produce a
value, so that it can be embedded in, say, the condition of a loop; and
perhaps somehow easing the burden of expressing big imports sets).  Base
Scheme is one of the less-terse of languages I know, and that took a
little getting used to, though eventually I decided it resulted in a
good balance of overall readability (and I could layer over the base
when that helped).  But maybe Racket2 decides to go more terse in the
base, which might be reasonable.  In any case, that's a bit of language
design space choices, which, in the context of Racket, I think should be
traced back to top-level requirements (e.g., why is the language being
made at all, for whom, for what purposes, and why that), and influence
much of the other requirements below it.  (I'm saying "requirements"
lately because it's a software engineering framework that should help us
be clear about things we haven't yet been, though I initially thought we
didn't need the formality.)

Simon Schlee

unread,
Sep 2, 2019, 9:49:46 AM9/2/19
to Neil Van Dyke, Racket Users
Am So., 1. Sept. 2019 um 22:55 Uhr schrieb Neil Van Dyke <ne...@neilvandyke.org>:
Simon Schlee wrote on 9/1/19 3:28 PM:
>
>     Try not to make identifiers be simple generic terms, like `watch`
>
>
> I strongly disagree with this statement.

I think what you say is valid, and I agree in some situations.

However, we're talking about reusable third-party packages, which are
used pretty much the same as much of the "standard library" of Racket.

Maybe the differences in our priorities stem from the differences in the way that we use racket.
I do not have the perspective of working on multi-person big code bases with racket and the special gotchas that come up with that.
Maybe that is one requirement that should be considered and talked about explicitly.
Consider creating an rfc/issue that describes some of the sticking points with current racket with big projects?
I think it would give valuable insight and allow me and other people to see problems from a different perspective.
I imagine that there is a big design space of solutions that would help for big projects and probably also rackety inventions that could still be made.

The reason why I find it appropriate to just call it `watch` is, because from my personal experience I find it highly unlikely that a file watching package would be used globally throughout an entire code base. It is much more likely that it will be used in 1-3 small modules and it is also likely that those modules will convey a general idea that they are all about handling files and the require will make it obvious that a file-watching library is used.
Especially if you are using DrRacket and its ability to show arrows to the require that provides that identifier.

I think we might not completely agree on the watch example, but it is good to see the different perspective and something to keep in mind while further thinking about it.

When it comes to worse is better, I do agree in the sense of getting things started and producing results in the short term. But I also think that it is valuable to sometimes take a break on short term solutions and consider what long term possibilities there could be. This might only result in a vague idea for future development or direction and you may end up needing the short term solution anyway, but I think its worth it to try to imagine different ways of doing things.
Reply all
Reply to author
Forward
0 new messages