package client proposal (a target for rock throwing)
Peter Michaux
than the modules subject. Kevin Dangoor thinks this is on subject for
this group. We need to start in at some point. I'll make a suggestion
based on my experiments in this area and see how it goes.
I imagine the package management client being installed first and then
used to install one or more JavaScript engines, standard libraries,
and community libraries. The following idea of multiple engines in one
repository doesn't depend on the package management client being
installed first but it does simplify the mental process.
Suppose there are just two libraries "print" and "greet" that we
standardize. The print library is engine specific (writing in C for
Spidermonkey, Java for Rhino, etc.). The greet library uses the print
library. The package repository would be something like the following
(note there are multiple versions of some packages)
packages/
index.json
spidermonkey_spidermonkey_1.8.jspkg
spidermonkey_spidermonkey_1.7.jspkg // just the spidermonkey code
rhino_rhino_1.7.3.jspkg
rhino_rhino_1.7.4.jspkg // just the rhino jar
core_spidermonkey_1.0.1.jspkg // "js" app with "require"
core_rhino_2.4.jspkg // "js" app with "require"
print_spidermonkey_1.4.jspkg // js and C
print_rhino_5.2.jspkg // js and jar
greet_all_1.1.jspkg
greet_all_0.9.jspkg
std_spidermonkey_0.4.jspkg // dummy package
std_rhino_0.4.jspkg // dummy package
Now on the client...
Suppose I want to create an installation of rhino and the standard
libraries
$ jspkg create rhinoinstall \
--architecture=rhino \
--rootdir=~/.rhino \
--repository=http://jspkg.mozilla.org/packages
creating ~/.jspkg/spidermonkeyinstall
$ jspkg rhinoinstall install std
getting index file from http://jspkg.mozilla.org/packages/index.json
installing rhino_rhino_1.7.4.jspkg
installing core_rhino_2.4.jspkg
installing print_rhino_5.2.jspkg
installing greet_all_1.1.jspkg
installing std_rhino_0.4.jspkg
(Note the above process is explicit and could be wrapped in a little
script that folks just run in a simple "./auto-install-rhino"
command.)
Then there might be files on my computer like
~/
rhino/
bin/
js
lib/
greet.js
print.js
print.jar
and I can do
$ ~/.rhino/bin/js -e="require('greet').greet('world');"
hello, world
I could then install spidermonkey and all the standard libraries
$ jspkg create spidermonkeyinstall \
--architecture=rhino \
--rootdir=~/.rhino \
--repository=http://jspkg.mozilla.org/packages
mkdir ~/.jspkg/spidermonkeyinstall
$ jspkg spidermonkeyinstall install std
getting index file from http://jspkg.mozilla.org/packages/index.json
installing spidermonkey_spidermonkey_1.8.jspkg
installing core_spidermonkey_1.0.1.jspkg
installing print_spidermonkey_1.4.jspkg
installing greet_all_1.1.jspkg
installing std_spidermonkey_0.4.jspkg
and do
$ ~/.spidermonkey/bin/js -e="require('greet').greet('world');"
hello, world
Note that greet_all_1.1.jspkg was installed twice: both in the rhino
and spidermonkey installs.
Note that multiple install libraries for a single engine can be
maintained with different versions.
Now throw rocks.
Peter
Robert Cerny
> Note that greet_all_1.1.jspkg was installed twice: both in the rhino
> and spidermonkey installs.
>
> Note that multiple install libraries for a single engine can be
> maintained with different versions.
>
> Now throw rocks.
Just for clarification: can i have multiple versions of one library
installed for one engine?
Robert
Kevin Dangoor
> The package management client and repository issues are more numerous
> than the modules subject. Kevin Dangoor thinks this is on subject for
> this group. We need to start in at some point. I'll make a suggestion
> based on my experiments in this area and see how it goes.
Great!
> I imagine the package management client being installed first and then
> used to install one or more JavaScript engines, standard libraries,
> and community libraries. The following idea of multiple engines in one
> repository doesn't depend on the package management client being
> installed first but it does simplify the mental process.
One other option is that if the package manager itself is written in
JS then you'd install an interpreter with some core set of libraries
and the package manager and pick up the rest via that. (Talk about
bootstrapping issues!)
But, I'm more inclined to imagine a package manager that is standalone
and can install the interpreter, as you suggest.
> Suppose there are just two libraries "print" and "greet" that we
> standardize. The print library is engine specific (writing in C for
> Spidermonkey, Java for Rhino, etc.). The greet library uses the print
> library. The package repository would be something like the following
> (note there are multiple versions of some packages)
>
> packages/
> index.json
> spidermonkey_spidermonkey_1.8.jspkg
> spidermonkey_spidermonkey_1.7.jspkg // just the spidermonkey code
I would like to go a step further here and suggest that it be
*possible* for the repository to have binaries in it. I don't know how
long it would take to compile spidermonkey on Windows, but I can
imagine it being a pretty unpleasant first time install experience if
the user has to wait for the interpreter to compile.
> rhino_rhino_1.7.3.jspkg
> rhino_rhino_1.7.4.jspkg // just the rhino jar
> core_spidermonkey_1.0.1.jspkg // "js" app with "require"
> core_rhino_2.4.jspkg // "js" app with "require"
> print_spidermonkey_1.4.jspkg // js and C
> print_rhino_5.2.jspkg // js and jar
Something else to consider: perhaps a package can be conceptually
split into a pure JS portion and an interpreter-specific portion.
Thinking about it, this seems like a premature optimization, but I
thought I'd mention it.
> greet_all_1.1.jspkg
> greet_all_0.9.jspkg
> std_spidermonkey_0.4.jspkg // dummy package
> std_rhino_0.4.jspkg // dummy package
When you say "dummy package", do you mean it's basically just a
package that lists other requirements?
> Now on the client...
>
> Suppose I want to create an installation of rhino and the standard
> libraries
>
> $ jspkg create rhinoinstall \
> --architecture=rhino \
> --rootdir=~/.rhino \
> --repository=http://jspkg.mozilla.org/packages
> creating ~/.jspkg/spidermonkeyinstall
I'm sure that you meant rhinoinstall there...
So, when you say "an installation of Rhino" you mean that this is the
command to get the Rhino interpreter?
> $ jspkg rhinoinstall install std
> getting index file from http://jspkg.mozilla.org/packages/index.json
> installing rhino_rhino_1.7.4.jspkg
> installing core_rhino_2.4.jspkg
> installing print_rhino_5.2.jspkg
> installing greet_all_1.1.jspkg
> installing std_rhino_0.4.jspkg
>
> (Note the above process is explicit and could be wrapped in a little
> script that folks just run in a simple "./auto-install-rhino"
> command.)
I would also think that you could have a convenience script so that
once you set up an interpreter environment, you can run a simple
command to install into that environment without setting rootdir or
architecture any more.
>
> Then there might be files on my computer like
>
> ~/
> rhino/
> bin/
> js
> lib/
> greet.js
> print.js
> print.jar
>
> and I can do
>
> $ ~/.rhino/bin/js -e="require('greet').greet('world');"
> hello, world
I also think we'd want to keep metadata about the installed packages
around (for uninstall, upgrades, etc.)
I also think that the ability to specify a "requirements" file for the
installer is really handy:
http://pypi.python.org/pypi/pip
> I could then install spidermonkey and all the standard libraries
>
> $ jspkg create spidermonkeyinstall \
> --architecture=rhino \
> --rootdir=~/.rhino \
> --repository=http://jspkg.mozilla.org/packages
> mkdir ~/.jspkg/spidermonkeyinstall
> $ jspkg spidermonkeyinstall install std
> getting index file from http://jspkg.mozilla.org/packages/index.json
> installing spidermonkey_spidermonkey_1.8.jspkg
> installing core_spidermonkey_1.0.1.jspkg
> installing print_spidermonkey_1.4.jspkg
> installing greet_all_1.1.jspkg
> installing std_spidermonkey_0.4.jspkg
>
> and do
>
> $ ~/.spidermonkey/bin/js -e="require('greet').greet('world');"
> hello, world
>
> Note that greet_all_1.1.jspkg was installed twice: both in the rhino
> and spidermonkey installs.
Makes sense.
> Note that multiple install libraries for a single engine can be
> maintained with different versions.
Do you mean that you'd install the engine into separate directories to
keep different versions of the libraries? (At least, I hope that's
what you mean... I've found that things can get hairy if you have one
library path with multiple versions of a library.)
> Now throw rocks.
Seems like a good start to me.
Kevin
--
Kevin Dangoor
work: http://labs.mozilla.com/
email: k...@blazingthings.com
blog: http://www.BlueSkyOnMars.com
Peter Michaux
The intention of the system is you cannot have multiple versions
installed for a single library set.
If module A requires B and C and they require different versions of D,
then problems can happen when both D's are loaded. There won't be a
namespace collision with the Securable Modules proposal but if D
really needs to be a singleton (e.g. database transactions, socket
connection, etc) then things break down. Unfortunately the developer
won't even know about it because he only knows he wants A and doesn't
know D is involved.
----
The real, intended solution is to maintain different library sets with
different versions.
$ jspkg create rhinoinstall \
--architecture=rhino \
--rootdir=~/.rhino \
--repository=http://jspkg.mozilla.org/packages
$ jspkg rhinoinstall install foo --version-1.0
$ jspkg create rhinoinstall2 \
--architecture=rhino \
--rootdir=~/.rhino2 \
--repository=http://jspkg.mozilla.org/packages
$ jspkg rhinoinstall2 install foo --version-1.1
Now when using the rhinoinstall library set you get version 1.0 of
foo. When using rhinoinstall2 library set you get version 1.1 of foo.
----
For a single library set, you can trick the system to install multiple versions.
$ jspkg create rhinoinstall \
--architecture=rhino \
--rootdir=~/.rhino \
--repository=http://jspkg.mozilla.org/packages
$ jspkg rhinoinstall install foo --version-1.0
$ jspkg rhinoinstall configure --rootdir=~/.rhino2
$ jspkg rhinoinstall install foo --version=1.1
Now make ~/.rhino/lib and ~/.rhino2/lib your PATH. There is no way to
select which version of foo will be used by require.
Peter
Wes Garland
How about a pebble: You need something in which to embed Spidermonkey, so that you can actually have your require() function.
Wes
--
Wesley W. Garland
Director, Product Development
PageMail, Inc.
+1 613 542 2787 x 102
Peter Michaux
>
> On Mon, Feb 9, 2009 at 9:00 PM, Peter Michaux <peterm...@gmail.com> wrote:
>> I imagine the package management client being installed first and then
>> used to install one or more JavaScript engines, standard libraries,
>> and community libraries. The following idea of multiple engines in one
>> repository doesn't depend on the package management client being
>> installed first but it does simplify the mental process.
>
> One other option is that if the package manager itself is written in
> JS then you'd install an interpreter with some core set of libraries
> and the package manager and pick up the rest via that. (Talk about
> bootstrapping issues!)
Yes the bootstrapping issues. The real mess comes with the desire to
make the packages that comprise that initial install also managed
through the package management system. Seeding the history of what has
been installed and what is to be removed by each package name is a
tricky part.
> But, I'm more inclined to imagine a package manager that is standalone
> and can install the interpreter, as you suggest.
It makes for a very clean code to install the package management client first.
The package management client can still be written in JavaScript. The
JavaScript interpreter used by the package management client would
just never be used by code other than the package management client.
>> Suppose there are just two libraries "print" and "greet" that we
>> standardize. The print library is engine specific (writing in C for
>> Spidermonkey, Java for Rhino, etc.). The greet library uses the print
>> library. The package repository would be something like the following
>> (note there are multiple versions of some packages)
>>
>> packages/
>> index.json
>> spidermonkey_spidermonkey_1.8.jspkg
>> spidermonkey_spidermonkey_1.7.jspkg // just the spidermonkey code
>
> I would like to go a step further here and suggest that it be
> *possible* for the repository to have binaries in it. I don't know how
> long it would take to compile spidermonkey on Windows, but I can
> imagine it being a pretty unpleasant first time install experience if
> the user has to wait for the interpreter to compile.
For whatever it is worth, I seem to remember compiling spidermonkey on
my Mac was pretty fast.
There is also the possibility of distributing a native windows binary
installer file that puts all compiled files in predetermined places.
This offers less control over the install locations but would be very
pleasant for the beginner.
>> rhino_rhino_1.7.3.jspkg
>> rhino_rhino_1.7.4.jspkg // just the rhino jar
>> core_spidermonkey_1.0.1.jspkg // "js" app with "require"
>> core_rhino_2.4.jspkg // "js" app with "require"
>> print_spidermonkey_1.4.jspkg // js and C
>> print_rhino_5.2.jspkg // js and jar
>
> Something else to consider: perhaps a package can be conceptually
> split into a pure JS portion and an interpreter-specific portion.
> Thinking about it, this seems like a premature optimization, but I
> thought I'd mention it.
>
>> greet_all_1.1.jspkg
>> greet_all_0.9.jspkg
>> std_spidermonkey_0.4.jspkg // dummy package
>> std_rhino_0.4.jspkg // dummy package
>
> When you say "dummy package", do you mean it's basically just a
> package that lists other requirements?
Yes that is what I mean.
>> Now on the client...
>>
>> Suppose I want to create an installation of rhino and the standard
>> libraries
>>
>> $ jspkg create rhinoinstall \
>> --architecture=rhino \
>> --rootdir=~/.rhino \
>> --repository=http://jspkg.mozilla.org/packages
>> creating ~/.jspkg/spidermonkeyinstall
>
> I'm sure that you meant rhinoinstall there...
Yes I did mean rhinoinstall there.
> So, when you say "an installation of Rhino" you mean that this is the
> command to get the Rhino interpreter?
Yes. Each "installation of Rhino" can go by some arbitrary label. I've
used "rhinoinstall" here. If there is a system wide install in
/usr/local then perhaps I would call my personal rhino install
"myrhino". It is just a label that can then later be used as an
argument to jspkg to manage a particular package set.
>> $ jspkg rhinoinstall install std
>> getting index file from http://jspkg.mozilla.org/packages/index.json
>> installing rhino_rhino_1.7.4.jspkg
>> installing core_rhino_2.4.jspkg
>> installing print_rhino_5.2.jspkg
>> installing greet_all_1.1.jspkg
>> installing std_rhino_0.4.jspkg
>>
>> (Note the above process is explicit and could be wrapped in a little
>> script that folks just run in a simple "./auto-install-rhino"
>> command.)
>
> I would also think that you could have a convenience script so that
> once you set up an interpreter environment, you can run a simple
> command to install into that environment without setting rootdir or
> architecture any more.
Configuration variables like rootdir and architecture are only set
when using the "jspkg create" command or "jspkg configure". After
that, using the package set label grabs those values which are stored
somewhere in the appropriate package set directory:
~/.jspkg/rhinoinstall, ~/.jspkg/myrhino, etc
>> Then there might be files on my computer like
>>
>> ~/
>> rhino/
>> bin/
>> js
>> lib/
>> greet.js
>> print.js
>> print.jar
>>
>> and I can do
>>
>> $ ~/.rhino/bin/js -e="require('greet').greet('world');"
>> hello, world
>
> I also think we'd want to keep metadata about the installed packages
> around (for uninstall, upgrades, etc.)
That is the primary reason for the ~/.jspkg directory. Each package
set has it's own directory there for storing information about that
package set.
> I also think that the ability to specify a "requirements" file for the
> installer is really handy:
>
> http://pypi.python.org/pypi/pip
sure, it could be
$ jspkg rhinoinstall install --packages=some/requirements.json
>> Note that multiple install libraries for a single engine can be
>> maintained with different versions.
>
> Do you mean that you'd install the engine into separate directories to
> keep different versions of the libraries? (At least, I hope that's
> what you mean... I've found that things can get hairy if you have one
> library path with multiple versions of a library.)
Yes that is what I mean. A particular use case for me is the Perl
modules on my development machine. I have different PERL5LIB paths
that I use for work and for play. (I play with Perl as little as
possible.) This is because work never updates their Perl modules but I
can have the newest versions for my own use.
Peter
Peter Michaux
>> Now throw rocks.
>
> How about a pebble: You need something in which to embed Spidermonkey, so
> that you can actually have your require() function.
That is the "core" module with has the "js" command line tool application.
Peter
Robert Cerny
> The intention of the system is you cannot have multiple versions
> installed for a single library set.
>
> If module A requires B and C and they require different versions of D,
> then problems can happen when both D's are loaded. There won't be a
> namespace collision with the Securable Modules proposal but if D
> really needs to be a singleton (e.g. database transactions, socket
> connection, etc) then things break down. Unfortunately the developer
> won't even know about it because he only knows he wants A and doesn't
> know D is involved.
Multiple versions of stateless modules should be no problem.
> For a single library set, you can trick the system to install
> multiple versions.
>
> $ jspkg create rhinoinstall \
> --architecture=rhino \
> --rootdir=~/.rhino \
> --repository=http://jspkg.mozilla.org/packages
>
> $ jspkg rhinoinstall install foo --version-1.0
>
> $ jspkg rhinoinstall configure --rootdir=~/.rhino2
>
> $ jspkg rhinoinstall install foo --version=1.1
>
> Now make ~/.rhino/lib and ~/.rhino2/lib your PATH. There is no way to
> select which version of foo will be used by require.
Yes, by helping the require function interpret the module identifier
correctly in a configuration file with the following info:
"if module A requires module B, do not use the one on the path, but
instead use this one"
Just an idea.
Robert
Wes Garland
That is the "core" module with has the "js" command line tool application.
Ah, I see. I thought you were implying that that was the spidermonkey js shell, which is also named js.
Also "core" might not be the best name if we want to encourage competing embeddings. "product name" would be more like it to me. A reasonable spidermonkey embedding should be doing more than just providing require() and serverjs-stdlibs IMHO.
> For whatever it is worth, I seem to remember compiling spidermonkey on my Mac was pretty fast.
Wes
Wes Garland
That is the "core" module with has the "js" command line tool application.
Peter -- one more thing to consider -- Spidermonkey shouldn't be built automatically by the package system, because it's very likely that any non-trivial embedding will want to customize the build.
Personally, I solve this problem by automating the building of spidermonkey alongside the building of the embedding.
Also the "core" module should have a way to pass configuration parameters into it. Like installation directory.
Maybe a quick question-answer language to go along with module installation, which result in a modified environ[] to the module's Makefile?
Wes
Brian LeRoux
authors are not going to write their code N times for every js vm out
there. Fragmented libs do not help the language at all. This is
exactly why no one project has emerged as the server js
implementation. Too much divergence between platforms and
implementations.
Kevin Dangoor
>
> I think this really illustrates the need for a shim project. Library
> authors are not going to write their code N times for every js vm out
> there. Fragmented libs do not help the language at all. This is
> exactly why no one project has emerged as the server js
> implementation. Too much divergence between platforms and
> implementations.
Actually, a big difference between the world that we're angling for
and the one that exists today is that any meaningful JS program will
be tied to its interpreter. WIth a rich stdlib and ecosystem of 3rd
party modules, there will be quite a few meaningful "pure JS"
libraries.
A way to conveniently wrap native code for multiple interpreters
simultaneously will definitely be a bonus, to be sure.
Kevin
--
Brian LeRoux
The scope of a stdlib these days, hell even C's, is rather large.
There will be huge divergence between implementations. Pure JS...
sure, I look forward to that too. In the meantime, as they say, the
great thing about standards is there are so many to choose from.
How long till a JS VM written in pure JS is the problem that needs solving? ;)
Peter Michaux
>
> I think this really illustrates the need for a shim project. Library
> authors are not going to write their code N times for every js vm out
> there.
coincidence more than some grand plan. If that coincidence can be
exploited then that is great. It cannot in the case of Rhino. I don't
know if it can be exploited otherwise. If some systems use C or C++
and have different model for how JavaScript objects work, then there
could definitely need to have a shim per library per VM. The main guts
of the library can be in a straightforward shared library.
> Fragmented libs do not help the language at all. This is
> exactly why no one project has emerged as the server js
> implementation. Too much divergence between platforms and
> implementations.
their favor. JavaScript is in an odd situation of many C
implementations.
Peter
Peter Michaux
>
>
> On Tue, Feb 10, 2009 at 10:42 AM, Wes Garland <w...@page.ca> wrote:
>>
>>> That is the "core" module with has the "js" command line tool
>>> application.
>
> Peter -- one more thing to consider -- Spidermonkey shouldn't be built
> automatically by the package system, because it's very likely that any
> non-trivial embedding will want to customize the build.
>
> Personally, I solve this problem by automating the building of spidermonkey
> alongside the building of the embedding.
Each embedding has it's own spidermonkey build? Is that really
desirable for one package set installed on a machine? It seems to me
that all apps in a package set should have the same spidermonkey
behavior and that would indicate one spidermonkey build.
> Also the "core" module should have a way to pass configuration parameters
> into it. Like installation directory.
I did something like this with my xpkg experiments. The package set's
configuration variables (in the examples above the "jspkg create" or
"jspkg config" are sent to the package's make file or before install
hook script. Then sed or something like it can update a Config.js
module so that code can inspect installation details.
> Maybe a quick question-answer language to go along with module installation,
> which result in a modified environ[] to the module's Makefile?
These can be part of an individual package's before install hook.
Peter
delisa mackey
Wes Garland
The fact that multiple JavaScript VMs are written in C (and/or C++) is
coincidence more than some grand plan. If that coincidence can be
exploited then that is great. It cannot in the case of Rhino. I don't
know if it can be exploited otherwise. If some systems use C or C++
and have different model for how JavaScript objects work, then there
could definitely need to have a shim per library per VM. The main guts
of the library can be in a straightforward shared library.
I've looked at v8 vs. spidermonkey, and actually -- it looks like any shim that could reasonably work to make *their* modules compatible could be also written for JNI.
So, Rhino isn't out of the question, but obviously, I see a much more complex shim than others in this group.
Wes
Kevin Dangoor
> I've looked at v8 vs. spidermonkey, and actually -- it looks like any shim
> that could reasonably work to make *their* modules compatible could be also
> written for JNI.
>
> So, Rhino isn't out of the question, but obviously, I see a much more
> complex shim than others in this group.
I think a shim for Rhino is not that big a deal. Java has such a large
collection of packages that you can do nearly anything without leaving
JVM bytecode.
mob
even understand the need for some native packages, but why does the
package library need to also distribute the language VM
implementations?
I would have thought that was the user's first choice - what
implementation they want. The would install the implementation and
then get the packages they want. It would be nice if the user could
use the packaging tool and not even have to specify what
implementation they are using. The package tool should be able to
figure that out.
Implementations are managed by a different group and revised on a
different schedule. Users need to be able to get those separately.
Also, they frequently come with the O/S or O/S packaging scheme. I
don't think we should be replicating that.
Michael
Peter Michaux
>
> I'm a bit confused. I understand packages containing script code and I
> even understand the need for some native packages, but why does the
> package library need to also distribute the language VM
> implementations?
The entire code base can be managed uniformly with the package
management client.
> I would have thought that was the user's first choice - what
> implementation they want. The would install the implementation and
> then get the packages they want. It would be nice if the user could
> use the packaging tool and not even have to specify what
> implementation they are using. The package tool should be able to
> figure that out.
There are several possible work flows for installing and getting
packages. Having the package manager first is not more difficult. It
is just different.
It is different because we are dealing with a world where there are
many implementations and we seem to be looking at housing those all in
the same package repostitory.
> Implementations are managed by a different group and revised on a
> different schedule.
That doesn't preclude the package management client being installed first.
> Users need to be able to get those separately.
They can with the package management client.
> Also, they frequently come with the O/S or O/S packaging scheme. I
> don't think we should be replicating that.
I think that is a separate issue. If some Debian maintainer wants to
make a .deb package that is their prerogative. Those OS-dependent
packages are obviously not platform-independent which is what we need.
Some operating system, the most popular ones, don't even have standard
package management systems.
Peter
mob
response.
On Tue, Feb 10, 2009 at 12:23 PM, mob <m...@embedthis.com> wrote:
> On Feb 10, 12:03 pm, Peter Michaux <petermich...@gmail.com> wrote:
>> It is different because we are dealing with a world where there are
>> many implementations and we seem to be looking at housing those all in
>> the same package repostitory.
>
> get SpiderMonkey, WebKit, V8, Rhino & Ejscript?
> Oops, forgot Opera.
This is what I was thinking. One master repository. I cannot see a
reason it wouldn't work and it cuts down on administration. The pure
JS modules don't need to be mirrored between all the different types
of repositories.
A particular company, could create its own repository and that could
also be searched by the package management client when installing
software.
> Is this a fork of all those packages and if not, how closely will it
> track the versions and update of all the engines.
Not a fork of the JavaScript engines. (That would be crazy!) Just
package the engines up and add them to the repository.
>> > Implementations are managed by a different group and revised on a
>> > different schedule.
>>
>> That doesn't preclude the package management client being installed first.
>
> complexity.
The JavaScript engines are not really packaged anyway. You can just
download the source somewhere.
In the case of Rhino it would take all of about 10 minutes to create a
new package for a new Rhino version.
>> > Users need to be able to get those separately.
>>
>> They can with the package management client.
>
> ruby first. Gem doesn't install Ruby - it uses ruby.
It is backwards to what we commonly see. That doesn't make it wrong
(or right) and it substantially cleans up the code in the packages
comprising the standard library. Try writing a distribution both ways.
When I was finally free of the idea that the package management client
comes second (like it does in ruby) I was amazed at how much cleaner
my code was.
>> > Also, they frequently come with the O/S or O/S packaging scheme. I
>> > don't think we should be replicating that.
>>
>> I think that is a separate issue. If some Debian maintainer wants to
>> make a .deb package that is their prerogative. Those OS-dependent
>> packages are obviously not platform-independent which is what we need.
>
>
> I'd like to see the package manager be written in JS pure and use the
> locally installed language.
> ie. the reverse way you are proposing. I think it is simpler
Not to implement.
> and works
> better with existing distribution mechanisms.
I don't see how it works better with existing distribution mechanisms.
I'm not saying I'm right. I'm just saying package management client
first is a clean solution so that everything is a package and
installed the same way.
Peter
mob
I spend a few nights two weeks ago knocking out a first pass of a
package manager for us. I looked a most of what was out there and
studied the issues a bit. I was just trying to get a package manager
for Ejscript -- not for all implementations -- so the task was much
easier. I suspended the effort when I saw what you were doing with
this effort.
On Feb 10, 2009, at 12:39 PM, Peter Michaux wrote:
> On Tue, Feb 10, 2009 at 12:23 PM, mob <m...@embedthis.com> wrote:
>
>> On Feb 10, 12:03 pm, Peter Michaux <petermich...@gmail.com> wrote:
>>> It is different because we are dealing with a world where there are
>>> many implementations and we seem to be looking at housing those
>>> all in
>>> the same package repostitory.
>>
>> Wow? Will this package repository be the master and preferred way to
>> get SpiderMonkey, WebKit, V8, Rhino & Ejscript?
>> Oops, forgot Opera.
>
> This is what I was thinking. One master repository. I cannot see a
> reason it wouldn't work and it cuts down on administration. The pure
> JS modules don't need to be mirrored between all the different types
> of repositories.
goal
is to bring server-side Javascript to be used pervasively for web
management for embedded. I love the idea of a repository of reusable
libraries and modules. So I'm a real supporter of this effort. But
I'm
not sure that putting another distributor of Ejscript in the chain
helps our customers. They will come back to us to the updates and
upgrades which may or may not be 100% compatible with the version in
the package repository. If a security issue comes up, we need to fix
it fast and having users depend on the repository to get the updates
-
test them - validate and then release, is not my first choice of a
distribution model.
>
>
> A particular company, could create its own repository and that could
> also be searched by the package management client when installing
> software.
>
>
>> Is this a fork of all those packages and if not, how closely will it
>> track the versions and update of all the engines.
>
> Not a fork of the JavaScript engines. (That would be crazy!) Just
> package the engines up and add them to the repository.
Could this be an option. ie. the package manager can fetch the
implementation but if already installed it just leverages it.
The downside of this is the chicken-and-egg. But the ideal is the
package manager can be written in JS.
>
>
>
>>>> Implementations are managed by a different group and revised on a
>>>> different schedule.
>>>
>>> That doesn't preclude the package management client being
>>> installed first.
>>
>> Agree, but taking on the redistribution of all the packages adds more
>> complexity.
>
> The JavaScript engines are not really packaged anyway. You can just
> download the source somewhere.
lot of polish onto Ejscript and we want users to be able to get it
without having to use the source.
> In the case of Rhino it would take all of about 10 minutes to create a
> new package for a new Rhino version.
lot more complex very quickly. Look at PHP and how complex it is to
build it. On windows, very few people can actually build it. All
the
native extensions and library dependencies make it a nightmare. Also
the windows tool chains are less standard. VS6, VS2000 VS.NET,
VS2005,
VS2008 and the express editions. Also the Windows SDKs are all over
the place. It is HARD to build complex packages on windows.
For embedded, you are typically cross compiling. This means non-
standard tool chains like arm cross compilers and also non-standard
libc (uclibc).
We also support VxWorks and other embedded O/Ss. I know this is not
the market that many folks consider, bit is is much larger than you
think. We already have embedded Javascript running in all IP phones
from Siemens, EMC storage devices, Juniper routers etc. It is the
hidden web.
>>>> Users need to be able to get those separately.
>>>
>>> They can with the package management client.
>>
>> This seems backwards to me. When I want to use Gem or Rails, I get
>> ruby first. Gem doesn't install Ruby - it uses ruby.
>
> It is backwards to what we commonly see. That doesn't make it wrong
> (or right) and it substantially cleans up the code in the packages
> comprising the standard library. Try writing a distribution both ways.
> When I was finally free of the idea that the package management client
> comes second (like it does in ruby) I was amazed at how much cleaner
> my code was.
>>>> Also, they frequently come with the O/S or O/S packaging scheme. I
>>>> don't think we should be replicating that.
>>>
>>> I think that is a separate issue. If some Debian maintainer wants to
>>> make a .deb package that is their prerogative. Those OS-dependent
>>> packages are obviously not platform-independent which is what we
>>> need.
>>
>> But this is the default way for languages in Linux.
>>
>> I'd like to see the package manager be written in JS pure and use the
>> locally installed language.
>> ie. the reverse way you are proposing. I think it is simpler
>
> Not to implement.
It came out fairly cleanly for me. What road blocks did you hit?
It may be easier for us as we've already got File, Http and other
library classes which made writing a package manager much easier.
>> and works
>> better with existing distribution mechanisms.
>
> I don't see how it works better with existing distribution mechanisms.
>
> I'm not saying I'm right. I'm just saying package management client
> first is a clean solution so that everything is a package and
> installed the same way.
One question, once the implementation is installed, it is a moot
point, right?
ie. your proposal will work both ways, either delivering the engine as
a package or using one that is already installed?
Michael
Peter Michaux
> On Feb 10, 2009, at 12:39 PM, Peter Michaux wrote:
>
>> On Tue, Feb 10, 2009 at 12:23 PM, mob <m...@embedthis.com> wrote:
> I can only speak for us. We focus on the embedded market and our
> goal
> is to bring server-side Javascript to be used pervasively for web
> management for embedded.
What kind of "embedded"? Devices like my toothbrush and clock radio?
(I don't mean that sarcastically.)
> I love the idea of a repository of reusable
> libraries and modules. So I'm a real supporter of this effort. But
> I'm
> not sure that putting another distributor of Ejscript in the chain
> helps our customers. They will come back to us to the updates and
> upgrades which may or may not be 100% compatible with the version in
> the package repository. If a security issue comes up, we need to fix
> it fast and having users depend on the repository to get the updates
> -
> test them - validate and then release, is not my first choice of a
> distribution model.
I think this is more of a business issue for your company. Your
company compared to the master repository could be like the Ubuntu
compared with Debian. Ubuntu has newer versions and maintains those
packages. Debian moves slower.
You could also host a mirror of the central repository and then have
another repository with fixes that haven't made it into the central
repository (and your mirror). This way you would have control of
everything and your customers could pull from your bug fix repository
before looking in your mirror.
>>> Is this a fork of all those packages and if not, how closely will it
>>> track the versions and update of all the engines.
>>
>> Not a fork of the JavaScript engines. (That would be crazy!) Just
>> package the engines up and add them to the repository.
>
> Whew. But you are still distributing them.
You are writing a whole JavaScript engine when several open source
engines exist!!!!! ;-)
> Could this be an option. ie. the package manager can fetch the
> implementation but if already installed it just leverages it.
> The downside of this is the chicken-and-egg. But the ideal is the
> package manager can be written in JS.
Writing the package manager in JS is an appealing option. I wanted to
do the same at the beginning. I can see the option of writing the
package manager in JavaScript winning.
>>>>> Implementations are managed by a different group and revised on a
>>>>> different schedule.
>>>>
>>>> That doesn't preclude the package management client being
>>>> installed first.
>>>
>>> Agree, but taking on the redistribution of all the packages adds more
>>> complexity.
>>
>> The JavaScript engines are not really packaged anyway. You can just
>> download the source somewhere.
>
> Beg to disagree. We are providing binary and source. We are putting a
> lot of polish onto Ejscript and we want users to be able to get it
> without having to use the source.
That isn't the case with Spidermonkey, V8, and WebCore as far as I
know. Those engines certainly have the highest name brand recognition
and I'd guess most people will want to play with them first.
>> In the case of Rhino it would take all of about 10 minutes to create a
>> new package for a new Rhino version.
>
> In the embedded space and when you get a lot of extensions it gets a
> lot more complex very quickly. Look at PHP and how complex it is to
> build it. On windows, very few people can actually build it. All
> the
> native extensions and library dependencies make it a nightmare. Also
> the windows tool chains are less standard. VS6, VS2000 VS.NET,
> VS2005,
> VS2008 and the express editions. Also the Windows SDKs are all over
> the place. It is HARD to build complex packages on windows.
>
> For embedded, you are typically cross compiling. This means non-
> standard tool chains like arm cross compilers and also non-standard
> libc (uclibc).
>
> We also support VxWorks and other embedded O/Ss. I know this is not
> the market that many folks consider, bit is is much larger than you
> think. We already have embedded Javascript running in all IP phones
> from Siemens, EMC storage devices, Juniper routers etc. It is the
> hidden web.
My personal guess is the work of this group is targeted to those
wanting to develop server-side web applications. I would also guess
that the embedded world is on the edge of that interest.
>>>>> Users need to be able to get those separately.
>>>>
>>>> They can with the package management client.
>>>
>>> This seems backwards to me. When I want to use Gem or Rails, I get
>>> ruby first. Gem doesn't install Ruby - it uses ruby.
>>
>> It is backwards to what we commonly see. That doesn't make it wrong
>> (or right) and it substantially cleans up the code in the packages
>> comprising the standard library. Try writing a distribution both ways.
>> When I was finally free of the idea that the package management client
>> comes second (like it does in ruby) I was amazed at how much cleaner
>> my code was.
>
> But you have to write it in C now? instead of JS.
I like C but don't really like the idea of writing the package
management client in C. It seems like such a silly thing to do. It
would be great if someone else did it! ;-)
Perl and Java are relatively universal and though depending on them
seems odd, the install needs to depend on something.
The problem with writing in JS is which JS engine will be used to run
the package management client? Where will that install go if multiple
installs are possible?
>>>>> Also, they frequently come with the O/S or O/S packaging scheme. I
>>>>> don't think we should be replicating that.
>>>>
>>>> I think that is a separate issue. If some Debian maintainer wants to
>>>> make a .deb package that is their prerogative. Those OS-dependent
>>>> packages are obviously not platform-independent which is what we
>>>> need.
>>>
>>> But this is the default way for languages in Linux.
>>>
>>> I'd like to see the package manager be written in JS pure and use the
>>> locally installed language.
>>> ie. the reverse way you are proposing. I think it is simpler
>>
>> Not to implement.
>
> Not sure I agree. I've had a go at this and while not near complete.
> It came out fairly cleanly for me. What road blocks did you hit?
I really like the idea that once the engine and package management
client are installed, everything can be upgraded, deleted, installed
through the package management client.
If the distribution of the engine and package management client come
together then there is a bit of a makefile mess. Its to be possible to
build both this combined distribution and also to make the packages to
put on the repository. Also when everything is installed in one go
like this, seeding the history of which packages "installed" which
files seems like error prone work.
There are other ways to bootstrap but they seemed even less appealing.
> It may be easier for us as we've already got File, Http and other
> library classes which made writing a package manager much easier.
>
>>> and works
>>> better with existing distribution mechanisms.
>>
>> I don't see how it works better with existing distribution mechanisms.
>>
>> I'm not saying I'm right. I'm just saying package management client
>> first is a clean solution so that everything is a package and
>> installed the same way.
>
> I can appreciate that.
>
> One question, once the implementation is installed, it is a moot
> point, right?
>
> ie. your proposal will work both ways, either delivering the engine as
> a package or using one that is already installed?
If the engine and package management client are installed together and
the appropriate files are created so that everything can be updated
through the package management client then yes, they are equivalent.
Peter
Tom Robinson
>
> On Tue, Feb 10, 2009 at 8:05 AM, Wes Garland <w...@page.ca> wrote:
>>
>>
>> On Tue, Feb 10, 2009 at 10:42 AM, Wes Garland <w...@page.ca> wrote:
>>>
>>>> That is the "core" module with has the "js" command line tool
>>>> application.
>>
>> Peter -- one more thing to consider -- Spidermonkey shouldn't be
>> built
>> automatically by the package system, because it's very likely that
>> any
>> non-trivial embedding will want to customize the build.
>>
>> Personally, I solve this problem by automating the building of
>> spidermonkey
>> alongside the building of the embedding.
>
> Each embedding has it's own spidermonkey build? Is that really
> desirable for one package set installed on a machine? It seems to me
> that all apps in a package set should have the same spidermonkey
> behavior and that would indicate one spidermonkey build.
Additionally, how many "embeddings" of the same interpreter do you
expect to have?
Right now the model for server side JS is to embed an interpreter in a
specialized "host" application, which is why every project listed here http://en.wikipedia.org/wiki/Server-side_JavaScript
is limited to a single JS engine and a single web server (except for
Jack... it's not a coincidence I'm bringing this up ;))
That's a turn off for me in terms of interoperability (i.e. there's no
way to use pieces of multiple projects, since they're each tightly
coupled to their "host"). It's one of the things I hope this project
will help change by giving developers all the tools they need within
the "standard" command line interpreters, rather than resorting to
project-specific embedding.
Am I on the right track, or does everyone else still plan on having
different host/embeds for different projects?
mob
> (I don't mean that sarcastically.)
players, military applications (don't ask), data acquisition, IP
telephony ... and the fun goes on. I should also add mobile. Ejscript
has been used in mobile feature phones as the basis of a widget
application framework.
This is not tiny embedded like a toothbrush, but devices and systems
with 32 CPUs running linux, FreeBSD, VxWorks etc.
> You are writing a whole JavaScript engine when several open source
> engines exist!!!!! ;-)
The reason is that embedded is a different execution profile with more
stringent footprint requirements. Existing engines like to stretch
their legs and have speed as the #1 goal. Our goal is minimal memory
without compromising functionality.
So while Ejscript is not one of the major Browser based
implementations, this is a very interesting market space for server
side javascript. The number of deployed units is very, very large.
(More devices than people ;-) We've been doing server-side
javascript for about 5 years now and have a good base of experience
providing web management applications using it.
> Writing the package manager in JS is an appealing option. I wanted to
> do the same at the beginning. I can see the option of writing the
> package manager in JavaScript winning.
HTTP support already as many do.
> That isn't the case with Spidermonkey, V8, and WebCore as far as I
> know. Those engines certainly have the highest name brand recognition
> and I'd guess most people will want to play with them first.
should expect it to look more like Rails and other frameworks and less
like a tool kit of bits.
> My personal guess is the work of this group is targeted to those
> wanting to develop server-side web applications. I would also guess
> that the embedded world is on the edge of that interest.
windows. The reason is all the native dependencies, libraries and tool
chain. This will be our future too as we add more and more native
code. This has nothing to do with embedded.
Regarding embedded being part of the domain -- I'll argue that is must
be included too. "Server-Side" doesn't mean Enterprise only. I think
we can incorporate the needs of embedded without splitting into 2
efforts.
>
> > But you have to write it in C now? instead of JS.
>
> I like C but don't really like the idea of writing the package
> management client in C. It seems like such a silly thing to do. It
> would be great if someone else did it! ;-)
>
> Perl and Java are relatively universal and though depending on them
> seems odd, the install needs to depend on something.
>
> The problem with writing in JS is which JS engine will be used to run
> the package management client? Where will that install go if multiple
> installs are possible?
can anyone else write a module that will run over all interpreters?
What's good for the goose ....
Wes Garland
> different host/embeds for different projects?
We have specific needs from the embedding platform, meaning we need to absolutely control it, but it is built so it can be general-purposed in many different ways.
I imagine that's what most embedding authors are doing, and it makes sense. What we all need to do is to stop re-inventing the wheel, though. Come up with binary and JS module loaders and common DSO entry points, and share our work. A portability layer or shim would be an added bonus, so we can share binary modules across embedding platforms. But that has the possibility of being a LOT of work, and not always desirable (performance).
Kevin Dangoor
> Am I on the right track, or does everyone else still plan on having
> different host/embeds for different projects?
In Pythonland, a lot of us have grown fond of "virtualenv" which lets
you have a separate Python environment for each app. That way, each
app gets a stable set of libraries to work in and you upgrade each app
as it's ready...
But, that's just one style of working...
And that's an independent thing... you don't really have an app
embedding the interpreter... you have something like mod_wsgi or
mod_rack managing interpreter instances.
mob
On Feb 10, 2:53 pm, Tom Robinson <tlrobin...@gmail.com> wrote:
> That's a turn off for me in terms of interoperability (i.e. there's no
> way to use pieces of multiple projects, since they're each tightly
> coupled to their "host"). It's one of the things I hope this project
> will help change by giving developers all the tools they need within
> the "standard" command line interpreters, rather than resorting to
> project-specific embedding.
>
> Am I on the right track, or does everyone else still plan on having
> different host/embeds for different projects?
standard and much stronger, people could use standard command line
interpreters and standard web hosting modules (or FCGI hosting)
without having to have as many custom embeddings.
The key is to get the foundation right with:
1. The library well specified and stronger providing the key APIs so
each user doesn't need to resort to native code all the time
2. Get a good package repository and solve the cross interpreter
issues. I think the key interfaces here are the package format itself,
the module public/require mechanism and a good optional native code
interface. All these issues seem to be getting good attention.
3. Respected hosting for the primary package repository
I think it is an open question as to how successful we will be doing a
x-implementation shim for a single native API. But even in the worst
case that we fail with this piece, if we succeed on the others -- we
all win as there will be a consistent user JS library, a respected
public repository and a body of packages to share.
If we do fail with a shim, then the implementations will just need to
do more work and provide the low level modules requiring native code
-- or module writers will need to write several variants of their
native module. Not ideal, but since the number of native modules will
eventually be a low percentage. This should not be a deal killer.
Michael