boilerplate generator not found Evergreen problem (1563/2939/1.6.0_11-b03/Windows 2000 5.0 Cygwin 1.5.25/x86 x1)
Martin Dorey
Chris reports:
(05:09:16 PM) Chris Reece: Well, I got an amusing "evergreen-boilerplate-generator: command not found" error
And belatedly tells me the rather important detail that it's Windows. I can reproduce it, every time, as you'd expect:
bash: c:\Program Files\software.jessies.org\evergreen\Resources\evergreen\bin\evergreen-boilerplate-generator: command not found
Chris Reece
> Chris reports:
>
>
> (05:09:16 PM) Chris Reece: Well, I got an amusing
> "evergreen-boilerplate-generator: command not found" error
>
>
> And belatedly tells me the rather important detail that it's Windows.
Ah, be fair. If I'd been reporting it, I'd have given you the full spec.
What I actually said was:
Chris Reece says: I'm not bothered enough taht I'm willing to join the
Googlegroup.
;)
Now, if I can repeat the "correct indentation" problem I had under
SPARC/Solaris 10 with Python source, *that* I'll report. And I've amended my
Googlegroup membership. But, as you suggested via IM, it doesn't seem to be a
general problem; I can't repeat it at work.
On Windows.
Elliott Hughes
that -- given that it's a script -- one machine has the appropriate
interpreter, and the other doesn't, and that this is the error you get if
the file's found, but its interpreter isn't?
--elliott
Elliott Hughes, http://www.jessies.org/~enh/
Chris Reece
bash-3.2$ which python
/usr/bin/python
bash-3.2$ python -V
Python 2.5.1
bash-3.2$ which ruby
/usr/bin/ruby
bash-3.2$ ruby -v
ruby 1.8.6 (2007-03-13 patchlevel 0) [i386-cygwin]
bash-3.2$ which evergreen-boilerplate-generator
/cygdrive/c/Program
Files/software.jessies.org/evergreen/Resources/evergreen/bin/evergreen-boilerplate-generator
Martin Dorey
The code in NewFileAction.java for choosing the boilerplate generator is not working as intended. evergreen-boilerplate-generator will always be on the path, thanks to subvertPath in invoke-java.rb. (Although that subvertPath code isn't very old, it does look older than this code, so I don't think the problems I'm writing up here are even partly a regression introduced by invoke-java.rb changes.) It's lucky that evergreen-boilerplate-generator is always on the path, or the toString call would die with a null pointer exception. Because this is never null, the "return" is dead code. I'm not sure what the intention was behind that conditional and indeed behind having the property at all.
Perhaps Elliott intention would be best fulfilled by moving evergreen-boilerplate-generator to lib/scripts/, which isn't on the path (so we'd use the new getScriptFilename to run it), explicitly if there's no evergreen-boilerplate-generator on the path (and no property, if we still want the property). One possible intention for the property is to give the user a way of disabling the default boilerplate. In that case, perhaps we should be testing the property for equality to "". A more obvious way of achieving the same end would be to have an evergreen-boilerplate-generator higher up the path that outputs nothing.
The most expedient way for me to fix both the bugs described above and the one below would be to always use the string "evergreen-boilerplate-generator" and to swap the two lines in invoke-java.rb that add components to the path. That would have the effect of moving the default evergreen-boilerplate-generator to the end of the path, allowing the user can override it once again.
I imagine that Elliott's response so far would include the suggestion that we should ditch subvertPath entirely. Although it's inadequately commented, one purpose that's currently serving is to facilitate finding the appropriate architecture's support binaries (but not shared libraries). That's currently being used for setsid and perhaps for some of the native Mac tools. We want to use our setsid on Windows even in the unlikely event that the user has installed Cygwin's util-linux package. We could solve that another way. One way would be to rename our program (again). I don't know why I hadn't learned the lesson that, if doesn't walk like a duck, then it shouldn't be called "duck". Anyway, my point here was really that we could avoid putting the application's "bin" directory, which actually contains scripts rather than binaries, on the PATH while still putting our support binaries, which are in architecture-specific directories, on the PATH.
If we want to move evergreen-boilerplate-generator to lib/scripts/, then I'll have to fix Chris's bug by translating a Unix style command line which contains Windows filenames (with their backslashes, drive letters and spaces - and we will have a space in the path to evergreen-boilerplate-generator) into something Cygwin bash will accept. That's probably impossible to get right in the general case. :(
Another option would be to change the signature of ProcessUtilities.makeShellCommandArray so that, instead of being passed a command line, it would be passed an array of arguments. That sounds like a deal of work but perhaps not too much and it would give us a place to do quoting right. The dubious quoting in eg CheckForLintAction.java hasn't bitten us, except perhaps on Windows. The callers that pass in Windows paths would have to wrap those strings with calls to something like FileUtilities.performCygwinRewrites (exactly like that function except performing the reverse translation). This feels like the high-quality solution to me, but it is a deal of work and it would inevitably suffer from incompleteness, when people forget to call the Cygwin-specific wrapper with their script paths.
private void fillWithInitialContents(File file) {
// FIXME: we could use FileType.guessFileTypeByFilename to allow the user to specify scripts that override a single type. YAGNI.
String defaultBoilerplateGenerator = FileUtilities.findOnPath("evergreen-boilerplate-generator").toString();
String boilerplateGenerator = Parameters.getParameter("boilerplateGenerator", defaultBoilerplateGenerator);
if (boilerplateGenerator == null) {
return;
}
Elliott Hughes
> The code in NewFileAction.java for choosing the boilerplate generator is
> not working as intended. evergreen-boilerplate-generator will always be
> on the path, thanks to subvertPath in invoke-java.rb. (Although that
> subvertPath code isn't very old, it does look older than this code, so I
> don't think the problems I'm writing up here are even partly a
> regression introduced by invoke-java.rb changes.) It's lucky that
> evergreen-boilerplate-generator is always on the path, or the toString
> call would die with a null pointer exception. Because this is never
> null, the "return" is dead code. I'm not sure what the intention was
> behind that conditional and indeed behind having the property at all.
property. the program i want to call is in a directory full of crap i
definitely don't want on my path, and it's too stupid to work when
called via a symbolic link.
i should probably fix it myself and use a symbolic link to it, but in
the meantime i need the property. (and it wasn't hard for me to
imagine that others might be in a similar situation.)
actually, what i could trivially do is plop a two-line bash script in
my bin that execs the real script. (phil would need his own copy,
though, where the value of the property would work for both of us. but
it's rare to be able to get rid of a property so easily.)
> Perhaps Elliott intention would be best fulfilled by moving
> evergreen-boilerplate-generator to lib/scripts/, which isn't on the path
> (so we'd use the new getScriptFilename to run it), explicitly if there's
> no evergreen-boilerplate-generator on the path (and no property, if we
> still want the property).
Cygwin reason for it to be there.
> I imagine that Elliott's response so far would include the suggestion
> that we should ditch subvertPath entirely.
> Although it's inadequately
> commented, one purpose that's currently serving is to facilitate finding
> the appropriate architecture's support binaries (but not shared
> libraries). That's currently being used for setsid and perhaps for some
> of the native Mac tools. We want to use our setsid on Windows even in
> the unlikely event that the user has installed Cygwin's util-linux
> package. We could solve that another way. One way would be to rename
> our program (again). I don't know why I hadn't learned the lesson that,
> if doesn't walk like a duck, then it shouldn't be called "duck".
> Anyway, my point here was really that we could avoid putting the
> application's "bin" directory, which actually contains scripts rather
> than binaries, on the PATH while still putting our support binaries,
> which are in architecture-specific directories, on the PATH.
stuff" didn't have the same ring, i guess. (when i first came across
"bin" i honestly assumed it was because that's where you chuck all
your crap.)
but i don't follow your argument. i think it's the application scripts
we want on the path and the support binaries (and support scripts) we
want hidden away and referred to by their full path. it's reasonable
for the user to assume that they can type "terminator" in a Terminator
window, say, even if they're on Mac OS or whatever where we don't
install anything in usual search path. (unlike Linux, where we do,
which is why i don't like stuff support scripts being in the
application bin --- they get installed to the system bin.)
(i could understand an argument that says we shouldn't put anything on
the path, and should rely on either packaging systems that let us
install on the existing path, or add functionality to our applications
to let Mac users and the like explicitly ask for that more invasive
kind of installation. i think that's on a to-do list somewhere, but
not currently using a Mac it's not a priority. [i do think using .pkg
on Mac OS is a non-starter. i don't know anyone who's happy telling
some crap they just downloaded off the net "yes, here's my
authorization: run arbitrary code as root".])
> If we want to move evergreen-boilerplate-generator to lib/scripts/,
given that i wrote it, and fairly recently, i can't think why it's in
bin/ to start with. it's not like the feelings expressed above are
particularly new ones. and all its moral equivalents are already in
lib/scripts/. what on earth was i thinking?
> then
> I'll have to fix Chris's bug by translating a Unix style command line
> which contains Windows filenames (with their backslashes, drive letters
> and spaces - and we will have a space in the path to
> evergreen-boilerplate-generator) into something Cygwin bash will accept.
> That's probably impossible to get right in the general case. :(
wasn't i just following the path of least resistance? (probably copy &
pasting from something more complicated like the man page to HTML
conversion code.) i can't see why we'd need the shell involved at all.
wouldn't
new String[] { boilerplateGenerator, file.toString() }
work fine? or do you really need to go via "cmd /c" on Windows? or was
it the "setsid" i really wanted?
given the process-oriented JNI we have in Terminator, i do sometimes
wonder whether we should just give up on Java's built-in Process stuff
and our myriad work-arounds. iirc, Terminator's PtyProcess already
calls setsid(2) and offers kill(2) and getpid(2). it already
implements Java-style streams. and it runs on every platform we
support. i suppose the only negative is that it would mean Evergreen
would be more reliant on Cygwin. but are we ever realistically likely
to escape that dependency?
--elliott
Martin Dorey
>> that we should ditch subvertPath entirely.
> +1
> it's reasonable for the user to assume that
> they can type "terminator" in a Terminator
> window, say, even if they're on Mac OS
> i don't like stuff support scripts being in the
> application bin --- they get installed to the
> system bin
> what on earth was i thinking?
> i can't see why we'd need the shell involved at all.
> do you really need to go via "cmd /c" on Windows?
> or was it the "setsid" i really wanted?
> Terminator's PtyProcess...
... would indeed solve part of the problem - it would happily exec a Cygwin script. It wouldn't help us cope with Windows filenames constructed by getScriptFilename or findOnPath, which is why I think we need to pollute all the calls to ProcessUtilities.
Martin Dorey
(Perhaps filling in the missing information is worth the risk of forking the thread when we were making good progress on the other branch.)
Not finding the interpreter and lack of executability were my first two thoughts too but the problem is actually that we're running:
"setsid", "bash", "--login", "-c", "C:\i\don't\remember\the\exact\path\and\reece\chopped\off\the\tail\of\the\email\thread\in\some\doubtless\well-intentioned\but\annoying\reminder\of\the\days\when\we\couldn't\afford\the\bandwidth\for\fulltext-unten\bin\evergreen-boilerplate-generator"
Cygwin bash doesn't like Windows paths, eg:
martind@mdorey ~
$ 'C:\cygwin\bin\ls'
bash: C:\cygwin\bin\ls: command not found
martind@mdorey ~
$ 'C:\cygwin\bin\ls.exe'
bash: C:\cygwin\bin\ls.exe: command not found
martind@mdorey ~
$
-----Original Message-----
From: evergre...@googlegroups.com
[mailto:evergre...@googlegroups.com]
On Behalf Of Chris Reece
Sent: Wednesday, April 01, 2009 18:44
To: evergre...@googlegroups.com
Cc: evergre...@googlegroups.com
Subject: Re: boilerplate generator not found Evergreen problem
(1563/2939/1.6.0_11-b03/Windows 2000 5.0 Cygwin 1.5.25/x86 x1)
I don't know about any of that, but:
Elliott Hughes
> > but i don't follow your argument
>
> That part of my argument related to the smallest change we could make to fix this. It wasn't a "we should do this" argument.
>
> >> that we should ditch subvertPath entirely.
> > +1
>
> But then -1 later with:
>
> > it's reasonable for the user to assume that
> > they can type "terminator" in a Terminator
> > window, say, even if they're on Mac OS
>
> So we'd be keeping the part of subvertPath that's contributing, in an innocent bystander kind of way, to the trouble here. Removing the other part, worthy though I think we'd agree it is, could be done independently.
independent of each other. on the subvertPath front, i really mean "at
most Terminator should be doing that". even then, as i perhaps
unclearly tried to say, i'm not keen on that. i just don't want to
punish Mac users. but how harsh a punishment is it, really? how many
users use the "terminator" script at all, as opposed to clicking an
icon somewhere? we could always add a FAQ "How do I use terminator(1)
on Mac OS" that tells them to put a symbolic link to .../
Terminator.app/Contents/Mac OS/Terminator (or whatever it is).
what i'm trying to say is that, although being able to type
"terminator" in a Terminator window on Mac OS is the reason to have
our application bins on the users' paths, i'm not convinced that's
sufficient reason. i'm +1 to not messing with the user's path even so.
if Mac OS really becomes a problem, i think i'd rather write some code
to sudo and make the link. (i believe TextMate, a Mac-only program,
does exactly this.)
> > i don't like stuff support scripts being in the
> > application bin --- they get installed to the
> > system bin
>
> Eh? Surely evergreen-boilerplate-generator doesn't get installed to the system bin. Oh yes it does. Good grief. What were you thinking? :)
another reply where i wonder aloud about GUI preferences versus
properties versus backquoting scripts...
i put it here to facilitate it being on the path. for most of our
backquoted scripts, we hard-code the default, because the default is
basically a string. here the default is still code; code that's
clearly expressed in a script. and "cp"ing that script is a good place
for would-be customizers to start with their own scripts. (if the
functionality were still in Java, they'd _have_ to translate it. as it
is, if they're happy using the same scripting language, they can use
it as-is.)
we could be cleverer, of course:
value = find-on-path()
if (!value) {
value = get-property()
}
if (!value) {
value = find-in-script-dir()
}
if no-one but me is overriding this at the moment, maybe just get-
property() and then find-in-script-dir().
at the same time, i've remembered why i call the boilerplate generator
via Bash rather than directly. what if you've set the property to
something like "autogen --brief"? (though that appears to be broken by
the quoting in the generated string.) FileIgnorer is actually the only
case i can see in Evergreen where we call makeShellCommandArray but
never need to (because the command contains no user-supplied part).
otherwise we seem to have correctly used String[] directly where
there's nothing fancy going on.
> > what on earth was i thinking?
>
> Perhaps "that there was some Cygwin reason why it had to be there", as you said earlier and as indeed, you could argue, there currently is. Using "evergreen-boilerplate-generator" and letting bash search the path would work fine. Putting a Windows path, like C:\blah\evergreen\lib\scripts\, on the front would stop it from working, like C:\blah\evergreen\bin\ is stopping it from working now.
>
> > i can't see why we'd need the shell involved at all.
>
> Windows won't run a script based on a Cygwin hash bang line. Even if we rename it to .rb, the best we're going to get is some Windows native installation of Ruby. That'd be OK for our particular purpose here, I suppose, but then we'd be dependent on having two installations of Ruby, which is just madness. Don't let's reopen the discussion about whether we could use Windows Ruby instead. We've already made everyone install Cygwin Ruby. We need to keep working with that. The most we're ever going to want Windows Ruby for is as an alternative, not a requirement.
use PtyProcess everywhere" argument.
> > do you really need to go via "cmd /c" on Windows?
>
> Ugh, no.
going back to Unix, i think we send too much stuff via the shell.
specifically, some calls to makeShellCommandArray seem bogus.
(especially, as you pointed out elsewhere, ones where we've just
manually looked up the path with findOnPath and then pass it straight
to makeShellCommandArray.) but having looked more closely at the
callers, i see this isn't as common a mistake as i used to think.
> > or was it the "setsid" i really wanted?
searching case. we have:
* our own scripts (we should know where they are, and go straight to
them).
* our own binaries (we should know where they are, and go straight to
them).
* our own libraries --- we already go straight to them.
* system binaries such as gnome-open(1) --- here it's right to rely on
searching the path.
evergreen-boilerplate-generator is (sort of) the first case. setsid is
the second case. i think we do the right thing for the other two cases
already. (we even try to undo the JVM's mangling of LD_LIBRARY_PATH!)
> That's useful for killing long-lived children and stopping ssh from asking for a password, neither of which are likely to be interesting for a boilerplate generator.
less path searching.)
> > Terminator's PtyProcess...
>
> ... would indeed solve part of the problem - it would happily exec a Cygwin script. It wouldn't help us cope with Windows filenames constructed by getScriptFilename or findOnPath, which is why I think we need to pollute all the calls to ProcessUtilities.
changing getScriptFIlename and findOnPath (or whatever survives the
reshuffle), because we know where they're going to be used. (we might
even enforce it for findOnPath via some kind of execvp equivalent. and
maybe only publicize isOnPath to allow code to only conditionally try
to run something.)
the only way things get used is by making the right code the simpler,
prettier choice.
--elliott
>
> read more »
Martin Dorey
> if no-one but me is overriding this at the moment, maybe just get-
> property() and then find-in-script-dir().
Properties resurgent? Sigh. Well, I look forward to the UI.
> i put it here to facilitate it being on the path. for most of our
> backquoted scripts
Is that just of historical interest or is that you saying, as it seems, that you want evergreen-boilerplate-generator left in /usr/bin?
> the only way things get used is by making the right code the simpler,
> prettier choice.
Laudable, I'm sure, but I'm really skeptical as to whether that's possible here. If we switch to a Cygwin execvp, then we go from requiring all of our script names to be translated to needing their arguments translated. That's probably a smaller problem and we don't then have to pass everything via the shell.
We could wrap the existing spawn and do our own hash-bang interpretation. That might get us somewhere, if we never need the shell. On which subject, this seems backwards:
> FileIgnorer is actually the only
> case i can see in Evergreen where we call makeShellCommandArray but
> never need to (because the command contains no user-supplied part)
In the case of, say, PerlDocumentationResearcher, we shouldn't be letting the user's input be interpreted as shell metacharacters. The user-specified part should be a single argument. CheckForLintAction would be safer if it didn't use the shell. The predefined checkers could easily be specified as String[] or we could split at space. We could assume the same for user checkers. If that inconvenienced anyone, then they could use an intermediary script. One of ManPageResearcher's two calls invites research into Little Bobby Tables' cousin Semicolon Remove Home Directory, something that we then have to go out of way to prevent:
// We don't just test for the "man:" prefix because we're going to send the rest directly to a shell, so we don't want any nasty surprises.
Pipelines would be a pain but we never seem to want anything more complicated than one pipe.
This is simple and not too ugly:
command += " \"" + FileUtilities.parseUserFriendlyName(filename) + "\"";
But also wrong. Still, it's interesting that we're wrapping our script argument. If it's buggered without wrapping on all platforms, then maybe it's not so unworkable, to have the wrapper do something Cygwin-specific, after all.
Elliott Hughes
> On Apr 1, 7:04 pm, "Martin Dorey" <mdo...@bluearc.com> wrote:
> > The code in NewFileAction.java for choosing the boilerplate generator is
> > not working as intended. evergreen-boilerplate-generator will always be
> > on the path, thanks to subvertPath in invoke-java.rb. (Although that
> > subvertPath code isn't very old, it does look older than this code, so I
> > don't think the problems I'm writing up here are even partly a
> > regression introduced by invoke-java.rb changes.) It's lucky that
> > evergreen-boilerplate-generator is always on the path, or the toString
> > call would die with a null pointer exception. Because this is never
> > null, the "return" is dead code. I'm not sure what the intention was
> > behind that conditional and indeed behind having the property at all.
>
> i've quite possibly never used the path code, because i use the
> property. the program i want to call is in a directory full of crap i
> definitely don't want on my path, and it's too stupid to work when
> called via a symbolic link.
>
> i should probably fix it myself and use a symbolic link to it, but in
> the meantime i need the property. (and it wasn't hard for me to
> imagine that others might be in a similar situation.)
>
> actually, what i could trivially do is plop a two-line bash script in
> my bin that execs the real script. (phil would need his own copy,
> though, where the value of the property would work for both of us. but
> it's rare to be able to get rid of a property so easily.)
path. at work i install the .deb and run from the Applications menu,
so i get the desktop's path. i've changed my .xprofile so next time i
log in (assuming i _can_ and haven't broken anything) that should be
fixed. but until then i still need the property.
i know you like calling out to scripts, and i know these aren't
insurmountable problems, but i think they show we don't have a good
answer to configuration questions yet.
at the moment, we have three kinds of configuration:
1. GUI
2. properties
3. backquoting a script
each has its own pros and cons:
GUI
+ most easily discovered
+ updates registered at an obvious time
- hardest to take away once people are using it
- no sharing(/overriding) across users on a particular machine(/site)
- we have no mechanism to express complicated preferences like
"default to X for all languages, but use Y for Java and Z for Bash"
properties
- hard to discover
- hard to edit (though that's easily fixed if we encourage people to
edit them in Evergreen)
- updates only registered at startup (though that would be easily
fixed)
+ easy sharing and overriding across users on a particular machine(/
site) [though i may have actually removed the #include equivalent, but
that's easily reinstated, and the lack of a "list" type ]
+ easy to express complicated preferences (just choose the relative
order of #includes and settings)
backquoting a script
- hard to discover
- hard to edit (fixable in the same way as properties, but not quite
as easily, since there's no one source)
+ updates registered at an obvious time
- relatively costly updates because we effectively "re-read" the
configuration each time.
+ best for sharing identical configurations if you can subvert other
users' paths
- but not as good as properties for sharing if you can't
- and overriding is quite awkward (i.e. it's awkward to mostly use
phil's settings for something but add one entry and remove another,
for scalars as well as lists)
there's another kind of overriding too, which properties handle best:
that's per-language settings. see CheckForLintAction for the canonical
example. yeah, you can write a dispatcher script that works out which
script should really be called, but i think per-language settings are
the more natural solution.
i'm starting to think that many of the problems with properties were
really defects of my implementation. not that i want to get rid of all
scripts: evergreen-boilerplate-generator should remain a script, for
example. but i'm not convinced it shouldn't be addressed via a
property.
i've been trying for some time to think of a comfortable way to move
all BlueArc's users (who're probably the _only_ users who're
benefiting from site-wide scripts) to properties pointing to those
scripts. but maybe i should start from the other end: add a properties
editor where changes take effect as soon as you apply them, and then
worry about the [then-]legacy direct calling of scripts.
maybe do what Bash and Emacs and Ruby and the like do: start by
reading configuration from some global place (non-existent by default)
and then read the user's configuration. if the administrator has
installed the relevant file, all the users who haven't set any
properties themselves get those settings. that would actually be the
most [or, indeed, only] convenient way for me to share with phil, too.
as for external tools, my first thought was that since they were
probably the other cause of the backlash against properties, they
should be... what? thinking about it more, i think the main problem
isn't so much that they're properties as that they're in the main
properties file and handled with the normal properties. i think it
would be better to have a directory [tree] for external tool files,
each of which may be XML or properties (i don't think it matters), and
none of which need to allocate themselves a magic number to avoid
accidentally overwriting a different external tool. each file is its
own namespace, if you like. (however implemented.) using a directory
lets us copy or link to external tool files from others (or, given a
directory _tree_, copy/link whole sets of tools; a set of Perforce-
specific tools, say, or a set of Ruby-specific tools), install them
from tar files or .debs or whatever, and even provide a default system
set.
i'm not saying "everything should be a preference". i think that kind
of blanket statement was where we went wrong last time. i'm not sure
of the exact rules of thumb, and hopefully if we're paying attention,
we'll work them out as we go along, but here are a few that spring to
mind:
personal preferences like fonts => GUI
(people expect to be able to trivially change these, and they're
probably the most likely changes people want to make. colors are an
awkward case because people also make use of color schemes, and those
do tend to be shared. but since we don't offer configuration of
colors, that's not worth losing sleep over until we do. the only
current GUI preference i'm not sure about is the list of extensions
not to index. but maybe GUI just needs to be slotted into a sensible
position in the overriding hierarchy. hard-coded defaults << site-wide
defaults << GUI << properties? merging GUI settings in anywhere is
pretty hard, though, and in the middle would be hardest of all. also,
maybe our properties need += as well as =, to implement the behavior
we currently have with echo-local-*-evergreen-should-not-index, but in
a way that lets people choose whether to extend or override.)
per-language preferences => properties with per-language overrides
(we could do this in the GUI, but i can't think of a really
comfortable way. and i think programmers -- our users -- are pretty
comfortable with the idea of textual if ... else if ... else ...
structures already; they just need to know the specific syntax.)
behavior => script, pointed to by properties with per-language
overrides (possibly with defaults)
(where actual computation is needed, we should run external code. that
way we let the users choose whether they want Bash or Ruby or even C++
or whatever as their implementation language. we should also expect
that users might have independent scripts for different languages, as
in CheckForLintAction.)
another mistake in the old properties code was reusing the system
properties. (this is part of why reloading was never implemented: it
was too hard.) we definitely shouldn't do that again.
i wonder how many people are using non-default properties? (and which
ones they're using.)
one thing none of these schemes really fixes is per-workspace
configuration. the configuration you want for work projects isn't
necessarily the same as for open source project 1, and open source
project 2 isn't necessarily the same as either. there are two
workarounds that have been suggested in the past. scripts can look at
their cwd (at the cost of being monolithic or needing dispatching
logic [which is itself just a small monolith]), and properties can use
dotfiles [up the directory tree, if you choose to group similar
projects, though that may mean fairly arbitrary groups in general, at
least it should keep "work projects" together] (the cost here being
dotfile cruft).
jobs, not necessarily in the right order:
* switch from system properties to a private map.
* implement reloading.
* add a FileAlterationMonitor for the properties file to automatically
reload.
* [re-]implement some form of #include (making sure the FAM picks up
changes to included files).
* implement some form of += (in addition to the existing =).
* choose a system-wide location and load that ahead of the user's
properties file (making sure the FAM picks up changes to the
"inherited" file).
* change hard-coded defaults to be a file in evergreen's lib directory
(loading at the right time and telling the FAM)?
* try to write One True Method to factor out the commonality in the
callers.
* choose a GUI (about:config-style: +discoverable +can offer per-
property help -lots of work -hard to implement per-language properties
well; simple dialog: +easy to implement per-language properties +not
much work -no help -weird to have a crappy editor embedded within a
good editor; just open the file: +easy to implement per-language
properties +very little work +richest editor -no help) --- probably
"just open the file".
orthogonally, for external tools:
* pull the external tools out of the properties file, breaking them
into one-file-per-tool in a directory structure.
* implement reloading.
* add a FAM for the directories in the structure to automatically
reload.
* add a GUI, preferably a proper Add/Edit/Remove dialog. (something i
keep wanting in various projects.)
additionally, if necessary:
* find out how many people are using custom properties (treating
external tools and "everything else" as two separate cases).
* write code to automatically transition between the old and the new
(where possible).
* i think our removal of properties has been pretty successful for
those things that should have been in the GUI, but we should keep an
eye out for configuration that's of the wrong kind.
* decide whether GUI settings need to be in the same hierarchy. if
they do, work out how to do that.
non goals that, if done, are separate projects:
* providing the ability to override system preferences.
* providing new preferences (of whatever kind) for things that aren't
currently configurable.
* providing a mechanism to set environment variables. (we could have
special syntax, but it's incompatible with the desire to be able to
undo [reload].)
* providing a general mechanism to backquote in properties files. (we
could do this, but i don't think it would help anyone. i might be
wrong, though, if someone needed to compute what most people consider
hard-coded. it would be awkward to do well [because you're hiding
changes from the FAM] so i'd want to hear someone have a specific use
case.)
--elliott
Elliott Hughes
> > if no-one but me is overriding this at the moment, maybe just get-
> > property() and then find-in-script-dir().
>
> Properties resurgent? Sigh. Well, I look forward to the UI.
> > i put it here to facilitate it being on the path. for most of our
> > backquoted scripts
>
> Is that just of historical interest or is that you saying, as it seems,
> that you want evergreen-boilerplate-generator left in /usr/bin?
reason, but that it's not convincing when thought about.
> > the only way things get used is by making the right code the simpler,
> > prettier choice.
>
> Laudable, I'm sure, but I'm really skeptical as to whether that's
> possible here. If we switch to a Cygwin execvp, then we go from
> requiring all of our script names to be translated to needing their
> arguments translated. That's probably a smaller problem and we don't
> then have to pass everything via the shell.
worst comes to the worst and we really need to say "this is a path,
this is text" for each argument, the builder style would let us do
something like .....new("ls").arg("-l").path("/etc").exec(), and hide
the mangling inside PtyProcessBuilder.path.
> We could wrap the existing spawn and do our own hash-bang
> interpretation. That might get us somewhere, if we never need the
> shell. On which subject, this seems backwards:
>
> > FileIgnorer is actually the only
> > case i can see in Evergreen where we call makeShellCommandArray but
> > never need to (because the command contains no user-supplied part)
> In the case of, say, PerlDocumentationResearcher, we shouldn't be
> letting the user's input be interpreted as shell metacharacters. The
> user-specified part should be a single argument.
> CheckForLintAction
> would be safer if it didn't use the shell. The predefined checkers
> could easily be specified as String[] or we could split at space. We
> could assume the same for user checkers. If that inconvenienced anyone,
> then they could use an intermediary script. One of ManPageResearcher's
> two calls invites research into Little Bobby Tables' cousin Semicolon
> Remove Home Directory, something that we then have to go out of way to
> prevent:
>
> // We don't just test for the "man:" prefix because we're going
> to send the rest directly to a shell, so we don't want any nasty
> surprises.
there's still room for accident. the user might not know what's on
their clipboard, say.
> Pipelines would be a pain but we never seem to want anything more
> complicated than one pipe.
>
> This is simple and not too ugly:
>
> command += " \"" + FileUtilities.parseUserFriendlyName(filename)
> + "\"";
>
> But also wrong. Still, it's interesting that we're wrapping our script
> argument.
> If it's buggered without wrapping on all platforms, then
> maybe it's not so unworkable, to have the wrapper do something
> Cygwin-specific, after all.
"wrapper").
don't we just want to switch to new String[] { boilerplateGenerator,
file.toString() }, use getScriptFilename instead of findOnPath, and
change getScriptFilename to processCygwinRewrites? (the last of which
should fix GoToTagAction, PythonDocumentationResearcher, and
RubyDocumentationResearcher on Cygwin too.)
EditServer should also stop using findOnPath, though i don't think we
have the necessary code to replace it with.
if you want to write up a concrete list of jobs, that would be
helpful, and let us divide the work (or at least make it clear who's
doing what/who needs what from whom).
btw, speaking of processCygwinRewrites, if we're going down the path
of using PtyProcess more widely, there's no reason not to add JNI for
the underlying function for that.
--elliott
>
> read more »
Martin Dorey
I'm not sure how to tell what properties I'm using but an attempted diff combined with knowing what I use every day suggests that I'm using three "tools" in my properties:
(1) a "bk edit" tool that's largely specific to how I personally have bk set up, but which I'd happily inflict on other BlueArc bk users
(2) a "shell window" tool that starts Terminator in $EDIT_CURRENT_DIRECTORY (how is that not there by default? ("EDIT"?)
(3) a "mail diffs" tool, which conveys the coloring, as well as the patch, to others
Ed Porter
On 4 Apr 2009, at 21:14, Martin Dorey wrote:
>
> (2) a "shell window" tool that starts Terminator in
> $EDIT_CURRENT_DIRECTORY (how is that not there by default? )
Is it not? I've got one, did I add mine indepedently?
-ed
Elliott Hughes
one of the mistakes we've made with properties in the past was that we
only had one properties file, and we used it to hard-code defaults _and_
for user configuration. we've moved away from that, but we never removed
or disabled the old files. so someone who's been using Evergreen since
before it was called Evergreen is likely to have all kinds of random
cruft.
for some time now, we've not been writing an "edit.properties" file at
all. so new users won't even have a file, let alone anything in it. if i
could actually rely on people to notice and/or mention it, an easy way to
find out what people are using would be to switch to looking for
"evergreen.properties" instead!
Ed Porter
On 4 Apr 2009, at 21:36, Elliott Hughes wrote:
>
> On Sat, April 4, 2009 12:30, Ed Porter wrote:
>> On 4 Apr 2009, at 21:14, Martin Dorey wrote:
>>> (2) a "shell window" tool that starts Terminator in
>>> $EDIT_CURRENT_DIRECTORY (how is that not there by default? )
>>
>> Is it not? I've got one, did I add mine indepedently?
>
> one of the mistakes we've made with properties in the past was that we
> only had one properties file, and we used it to hard-code defaults
> _and_
> for user configuration.
That'll be where mine came from then.
When I updated Evergreen for the first time in ages the other day, I
was surprised to find my 'rebuild everything' command which has been
attached to ctrl-shift-b for longer than I can remember didn't work;
it had been hijacked by a command which failed in my environment
('make test').
User-defined keyboard actions should take precedence over hard-coded
defaults.
-ed
Martin Dorey
> or mention it
Sod em. Can't make omelet without breaking eggs.
Sent: Sat Apr 04 12:36:19 2009
Subject: Re: why i still sometimes use properties (was Re: boilerplate generator not found Evergreen problem (1563/2939/1.6.0_11-b03/Windows 2000 5.0 Cygwin 1.5.25/x86 x1))
On Sat, April 4, 2009 12:30, Ed Porter wrote:
> On 4 Apr 2009, at 21:14, Martin Dorey wrote:
>> (2) a "shell window" tool that starts Terminator in
>> $EDIT_CURRENT_DIRECTORY (how is that not there by default? )
>
> Is it not? I've got one, did I add mine indepedently?
one of the mistakes we've made with properties in the past was that we
only had one properties file, and we used it to hard-code defaults _and_
Elliott Hughes
> My reason for liking the scripts is the success of the site-wide
> configuration at BlueArc. I'm pretty sure Evergreen wouldn't have "taken
> off" to the degree that it has (even to new starters in the UK office,
> with no evangelists) without either a lot of BlueArc cruft in the
> Evergreen code or changes to the BlueArc codebase that would have been
> hard to justify... or good site-wide support for the ignore stuff.
actually trying to move forward.
> Anything that isn't a step backwards from where we are now, as far as
> that is concerned, is fine by me.
always stay in. a script to create a .deb from a configuration file is the
best i can think of. Emacs and Vim users are used to having to explicitly
import some company-specific configuration, but it would be nice to do
better [without resorting to path shenanigans].
> I'm not sure how to tell what properties I'm using but an attempted diff
> combined with knowing what I use every day suggests that I'm using three
> "tools" in my properties:
>
> (1) a "bk edit" tool that's largely specific to how I personally have bk
> set up, but which I'd happily inflict on other BlueArc bk users
>
> (2) a "shell window" tool that starts Terminator in
> $EDIT_CURRENT_DIRECTORY (how is that not there by default? ("EDIT"?)
it. (opening in the workspace root would have been more useful to me.)
"EDIT" because, again, i didn't want to break existing stuff by switching
to EVERGREEN. see also "edit.preferences". maybe this is a good time to
change both: when only old-timers have these files or anything useful in
them.
> (3) a "mail diffs" tool, which conveys the coloring, as well as the
> patch, to others
properties file. i think most of the other stuff that still works (as
opposed to random cruft that people might have left lying around from
2001) is not-yet-documented per-language stuff.
--elliott
Elliott Hughes
yes, or there should at least be a warning. there's a to-do, but external
tools see such light use (probably because there's neither UI nor
documentation, and you have to restart Evergreen for changes to take
effect) that it's never been a priority.
Martin Dorey
> best i can think of
Martin Dorey
> don't we just want to switch to new String[] { boilerplateGenerator,
> file.toString() }
You had what I understood to be a hypothetical counter-example yesterday, where the property was specifying a bash command line rather than the name of a script. That apparently didn't work anyway for some quoting reason. Can I assume that we ain't gonna need that? The user can always have an intermediary script to run the command line for them. If so, then I think we can solve the Cygwin boilerplate issue without the disruption of a new process builder, though I do like your suggested syntax and it's not without other motivation, agreed upon below.
Perhaps I should move the finding of the interpreter to ProcessUtilities next and before touching NewFileAction, so that NewFileAction doesn't require an explicit "ruby" and hence the same FIXME/caveat as I just added to GoToTagAction.
Elliott Hughes
>> don't we just want to switch to new String[] { boilerplateGenerator,
>> file.toString() }
>
> You had what I understood to be a hypothetical counter-example
> yesterday, where the property was specifying a bash command line rather
> than the name of a script. That apparently didn't work anyway for some
> quoting reason.
me think my suggestion above was fine for now. but, yes, in the longer
term it's sub-optimal.
> Can I assume that we ain't gonna need that? The user can
> always have an intermediary script to run the command line for them.
anything more yet. generally, no. CheckForLintAction is a good example
where i've yet to meet a tool that doesn't require some arguments to
behave sensibly.
we can trivially reproduce Sun's Process' behavior by using a
java.util.StringTokenizer (that's all Runtime.exec does). sadly, that's
pretty shite and not really what you want. it doesn't even try to
understand quoting, for example. we may as well use cmd.split("\\s"). but
i think that's fine for now.
maybe my desire to call the shell less is a mistake, though. we're always
going to involve the shell in external tools, for example.
however we do it, i do think we should avoid forcing the user to create
special scripts to work around Evergreen's handicaps.
> If
> so, then I think we can solve the Cygwin boilerplate issue without the
> disruption of a new process builder, though I do like your suggested
> syntax and it's not without other motivation, agreed upon below.
future. but we should fix chris' immediate problem sooner than that.
> Perhaps I should move the finding of the interpreter to ProcessUtilities
> next and before touching NewFileAction, so that NewFileAction doesn't
> require an explicit "ruby" and hence the same FIXME/caveat as I just added
> to GoToTagAction.
where it's our own implementation detail that the user can't override, but
not in general. i don't think reading the file before executing it is a
direction worth investing any effort in.
Martin Dorey
> i don't think reading the file before executing it
> is a direction worth investing any effort in.
> we should fix chris' immediate problem sooner than that.