The lifecycle of develop/build/submit/install/run in Newspeak
47 views
Skip to first unread message
Milan Zimmermann
Jul 7, 2021, 7:43:59 PM7/7/21
to Newspeak Programming Language
Hi,
Having kept extending this post so much, I should say at the beginning and then at the end again: "Thanks and sorry for this long on context and questions, but light of facts post" :)
My intention is to experiment with, and write up something about the lifecycle of packaging, distributing, deploying and installing applications and libraries in Newspeak.
I realize much of this has been discussed, and described in an earlier Gilad's post here:
My main reason for a follow up is to clarify the broader context for myself - which will surely reveal a level of continued ignorance :) . Partly my internal need for clarificaions is maybe due to some workflow and terminology tension or overlap in 'mainstream' software distribution compared to Newspeak.
There are two sections here, A) and B). My purpose of including section A) is ONLY to create context of how today's 'mainstream' software deals with packaging, distributing, deploying and installing applications and libraries, and examine differences of the (to be better understood by me) Newspeak lifecycle in section B).
A note - I already went over parts of such lifecycle in Newspeak during the steps of building and local-server deployment of CounterApp.vfuel described here: https://mzimmerm.github.io/2021/05/newspeak-a-few-notes#org2fffd3b. But I have not actually dug into details of packaging and building methods there - hence this long post.
A) The lifecycle of develop/build/submit/install/run an application, described for the current "mainstream" way of software distribution using package managers and package repositories
---------------------------------------
Feel free to ignore this section A. It's for context only and it grew out of proportion :) Although there is a reference to the lifecycle in section B)'s questions.
Let me start with my understanding of the current "mainstream" way of software distribution using package repositories. I'd describe such lifecycle as follows:
A1) 'develop' : 'developer' 'develops' the 'program' or 'app'.
A2) 'build' : 'developer' 'builds a 'program package' from the 'program' and (potentially) it's 'dependencies', using a 'build tool'. Such 'build tool' is generally part of a higher level 'package manager' - the 'package manager' may act in 2 steps, in this build step, and in the 'install' step 4. The build tool may be gradle, rpm builder, docker image builder, snapcraft, flatpak builder etc. The resulting 'program package' may be a war file, a vfuel file, an rpm file, etc. The 'program package' may or may not be 'self-contained' (contain all needed components - dependencies - to run the program). Either way, 'dependencies' (libraries needed to run the program) must be resolved either at this build step (if the 'program package' is self-contained) or at the 'target system' 'install' step. As an example, the war or snap file would contain all dependencies, while the rpm file would rely on dependencies to be resolved and installed during the 'install' step.
A3) 'submit': 'developer' 'submits' the 'program package' to a 'package repository'. On the web, developer copies the jar or vfuel to a webserver (so the repository is skipped? unless the browser extensions store is used)
A4) 'install': 'end-user' 'installs' the 'program' on the 'target system' from the 'program package' located in the 'package repository', using a 'package manager'. As discussed in step 2, the 'package manager' of the 'non-self-contained' program packages must calculate and install 'dependencies'.
A5) 'run': 'end-user' 'runs' the 'program' installed in step 4.
Notes:
- steps A1, A2 and A3 (develop, build, submit) generally are executed - or at least can be - on the 'source system' (where the software is developed).
- step A3 (submit to a repository) copies the 'program package' to a publicly available repository (available to both developer and end-user).
- steps A4 and A5 (install, run) are executed on the 'target system' - where the end-user installs and runs the program. Well, unless we are on the web where the 'end-user' points browser to a URL.
- I tried not to use the verb "to package" here (only "to build"). Unless I am missing something, "to package a program" is pretty much equivalent of "to build a package from a program" - so "to build" == "to package". These actions happen on the 'source system'.
- The difference between 'self-contained' program packages and the not-self-contained ones, and how they are handled by their 'package manager' programs, is similar to early and late binding, perhaps with somewhat similar pros and cons.
B) What are roles of the methods (verbs) 'build' and 'package' in the Newspeak lifecycle of develop/build/submit/install/run an application?
---------------------------------------
Grepping the Newspeak codebase for:
find . -type f -exec grep -H "package.*Using.*:" {} \;
Yields the following methods signatures (on code from like a month ago when I started writing this):
- SomeClass.ns: #packageUsing: manifest = ..
- SomeTestsConfiguration.ns: #packageTestsUsing: manifest = .. (or ideNamespace but that's just a name?)
In addition, in the post
Gilad defines
- #packageLibraryUsingManifest: manifest
but this is not used yet anywhere in code.
So we have 3 "#package: manifest" type methods.
I think my first few questions would be:
- Is there a convention around the #packageTestsUsing:, in terms of which class it should be defined in, and invoked (sent) in? I see it generally defined in names like SomeTestsConfiguration.ns. I think the IDE uses this method's presence to show the [runTests] button(?). Are there othe objects invoking this method , e.g. if developers want to include tests in the exported package?
- What is the reason for the 3 method signatures, would perhaps 2 signatures , "#packageUsing: manifest" and "#packageTestsUsing: manifest" suffice? I am guessing there is a role of #packageTestsUsing: if the packager want to include tests. But what different roles do #packageLibraryUsingManifest: vs. #packageUsing: support? If we want to package an app (which I guess must contain a method #main: ) vs. a library, would the presence of #main: suffice to distinguish?
Moving on from there, I will focus on the example Gilad gave in the earlier linked post in this group in https://groups.google.com/g/newspeaklanguage/c/kHAIE_i7gTc/m/2KwHVGIFBQAJ
Rephrasing that post slightly:
```
Imagine a convention whereby *every library intended for distribution is sent out as a class which:*
- has a factory (or in general, class method) method ~#packageLibraryUsingManifest: manifest~ -
- has a ~#build: platform~ method - this method, given a platform object, produces a working instance of the module we actually want to distribute
```
(NOTE: Below, I am using abbreviated method name "#package:" instead of "#packageLibraryUsingManifest:" or "#packageUsing:" )
My main reason for including the long section A) above (the section tried to describe the current 'mainstream' package lifecycle and package managers) was to elaborate on the roles and differences between the "#build: platform" and the "#package: manifest" methods.
I realize I could and should build better understanding by studying the code, but wanted to start with context.
So I have some initial questions about the Newspeak lifecycle of develop/build/submit/install/run an application. I realize the questions might not always make sense or out of context, but here they are:
- "#package: manifest" questions:
- At which step of the lifecycle would the "#package: manifest" method be used (invoked)? I think it is on the 'source platform', and so invoking this method is equivalent to the 'build' step in item A2 section A?
- What is the result of invocation of "#package: manifest"? Is it an object that is intended to be possibly streamed in a file for distribution - thus equivalent to a 'program package'. If or how does invoking this method result in the 'vfuel' file? (this is expanded in the next question below)
- Is the general intent that the tools (the IDE) invokes this method upon clicking the [deploy] -> #asVictoryFuelWithMirrors?
- If true, what are other high level steps for the .vfuel file to be produced (which I know is produced, I went through it in the CounterApp, just not quite grasping the steps even on a high level :)
- I think we can consider the vfuel file the 'program package' discussed in A2, A3, A4, but am I right?
- Can the 'program package' in Newspeak be something else than a vfuel file? I can place .vfuel under a webserver and run the CounterApp, but can I, for example, export some other artifact (zip file with binary or source?) that is later loaded into another persons Newspeak IDE?
- "#build: platform" questions (I think I sort of grasp the #package but not the #build):
- At which step of the lifecycle would the "#build: platform" method be invoked? Would it be on the 'target system', making "#build: platform" equivalent to the 'package manager' doing the 'installation step' equivalent [item A4 in section A]?
- If so, how and in which format would the result of the invocation of "#package: manifest" be "shipped" to the 'target platform'?
- Is the general intent that the tools (the IDE) invokes this method? If so (as we do not yet have an example), could you briefly describe how that would work, for example "after downloading the 'program package', in the IDE on top, we will have a button 'install package'" - well, I am being presumptious :)
A bonus question : )
What makes a Newspeak class to show the following buttons: [deploy] [configuration] [run] [debug]? I am making an assumption that:
- [deploy] - shows if the class has the "#package: manifest" type method?
- [configuration] - not sure what makes this show, perhaps class name ending with Configuration?
- [run] and [debug] - shows if the class has "#main: platform args: args" method?
- [runTests] - shows if the class has " #packageTestsUsing: manifest" method?
Can we please correct this and or elaborate?
Thanks and sorry for this long on questions and context, but light of facts post
Milan
Gilad Bracha
Jul 8, 2021, 12:41:47 AM7/8/21
to newspeak...@googlegroups.com
Hi Milan,
I'll answer the specific questions below. But first, let me give some background. This has been covered elsewhere, e.g., in section 2.3 of
but not necessarily in the most accessible way.
In Newspeak, everything is an Object, and that naturally includes applications. By convention, an application is an object with a method #main:args: that starts the application. This is not all that different from a C program binary, which is really and object with a method main(). The terminology differs a.out, and no one thinks at the proper level of abstraction, but a.out is an object all the same. So, by the way, is anything that supports an API, which is really just a set of named functions that operate on some encapsulated data - in other words a procedural interface to an object.
Being a class based language, the application object has a class which defines the application behavior. By convention, we expect that class to be instantiated by calling #packageUsing:. Instantiating the application class is analogous to building the application. The argument to the factory is the manifest, which is a namespace object where all the dependencies of the application can be found. The factory method of the application is analogous to its build script.
To deploy an application, you serialize it (exactly in the sense of object serialization) so that you can then transport it elsewhere, where you can deserialize it, obtaining a live object; then you call #main:args: and run it. Again, it's just like shipping executable. Deserializing is like loading.
Note that #main:args: expect two arguments - a Newspeak platform object and a list of "command-line" arguments (argv, in C parlance). The platform object reifes the set of standard libraries that one expects to have at any destination.
In a sane world, this would be the complete story. Alas, we live on Earth instead. We can't expect every device to have Newspeak installed. Therefore, we may need to package an application with additional elements, such as the Newspeak platform and a Newspeak VM. We also need to accommodate different packaging formats and technologies, such as operating systems and web browsers.
On Wed, Jul 7, 2021 at 4:44 PM Milan Zimmermann <milan.zi...@gmail.com> wrote:
I think my first few questions would be:- Is there a convention around the #packageTestsUsing:, in terms of which class it should be defined in, and invoked (sent) in?
Yes. This is a separate topic, with relatively little relationship to your main point. It has nothing to do with deployment. The full story is in the class comment of class Minitest, though perhaps slightly out of date.
The idea is that test suites can be run in different configurations. Since Minitest needs to manipulate test configurations generically, it needs them to conform to a uniform interface. Part of that interface is #packageTestsUsing:, which
allows Minitest to instantiate test configurations uniformly. At the same time, it provides the configuration with access to a complete namespace, so any configuration can find what it needs. Minitest can then call the configuration instances to run the tests, using #testModulesUsingPlatform:minitest:.
I see it generally defined in names like SomeTestsConfiguration.ns. I think the IDE uses this method's presence to show the [runTests] button(?).
Yes, it uses it to produce the [run tests] and [show tests] buttons. As noted above, it is also used by Minitest (the unit testing framework) to run the tests.
Are there othe objects invoking this method , e.g. if developers want to include tests in the exported package?
Asking for senders of #packageTestsUsing:, I don't see any outside of the IDE itself. Of course, you are free to call it.
- What is the reason for the 3 method signatures, would perhaps 2 signatures , "#packageUsing: manifest" and "#packageTestsUsing: manifest" suffice? I am guessing there is a role of #packageTestsUsing: if the packager want to include tests.
No, as I said, it is for test configuration.
But what different roles do #packageLibraryUsingManifest: vs. #packageUsing: support? If we want to package an app (which I guess must contain a method #main: ) vs. a library, would the presence of #main: suffice to distinguish?
Maybe. We never implemented #packageLibraryUsingManifest:. It does make clear what's an app and what's a library.
<snip>
- "#package: manifest" questions:- At which step of the lifecycle would the "#package: manifest" method be used (invoked)? I think it is on the 'source platform', and so invoking this method is equivalent to the 'build' step in item A2 section A?
The method is called #packageUsing: and, yes, invoking it is somewhat analogous to building an application.
- What is the result of invocation of "#package: manifest"? Is it an object that is intended to be possibly streamed in a file for distribution - thus equivalent to a 'program package'. If or how does invoking this method result in the 'vfuel' file? (this is expanded in the next question below)
The result is a Newspeak object that constitutes the application. It is distinct from an object for purposes of distribution. There are many possible distribution formats. The "package" in the sense of an entity to be distributed, is the result of serializing the application object. One can serialize the object in different ways, corresponding to different distribution formats and different assumptions about dependencies. The intent is that the serialization is performed via the deploy options in the IDE.
- Is the general intent that the tools (the IDE) invokes this method upon clicking the [deploy] -> #asVictoryFuelWithMirrors?
Yes.
- If true, what are other high level steps for the .vfuel file to be produced (which I know is produced, I went through it in the CounterApp, just not quite grasping the steps even on a high level :)- I think we can consider the vfuel file the 'program package' discussed in A2, A3, A4, but am I right?
Yes, you can view a vfuel file as a particular packaging/distribution format.
- Can the 'program package' in Newspeak be something else than a vfuel file?
Yes. Example: you compile the application into Javascript. This too is a way to serialize the application object. Or you might want to deploy your code as an Electron app, packaged as a MacOS application or a Window application, and these may each have variations (are they to be signed for app store use?; are they to include the Newspeak code as vfuel along with a WASM VM, or as Javascript? etc.). More on this below.
I can place .vfuel under a webserver and run the CounterApp, but can I, for example, export some other artifact (zip file with binary or source?) that is later loaded into another persons Newspeak IDE?
In principle, yes, there are other ways to serialize an application. In earlier versions, we would serialize an app without the platform code. This results in very small files (say 100Kb, depending on the app). The IDE (this was the old Smalltalk based system) would deserialze the app object into a regular Newspeak object in memory. In an ideal world, this is all you would need. Alas, this requires a Newspeak engine to be useful, and we cannot realistically assume this.
So, the current vfuel files include the entire platform code in them. Now, since many apps do not require features such as reflection, and reflection tends to greatly increase the footprint of the vfuel file (because we must include source code as well as the mirror module) we have options to create vfuels with or without mirrors. The idea of deployment configurations is to generalize this, so one can describe deployments of varying kinds. This is not really properly supported yet. But you can see that there is a large number of combinations (JS or WASM; GUI or non-GUI; mirrors or no mirrors; MacOS, Windows, ChromeOS, iOS, Android; Electron or PWA) not all of which are valid of course.
- "#build: platform" questions (I think I sort of grasp the #package but not the #build):- At which step of the lifecycle would the "#build: platform" method be invoked?
When loading an application.
So #build: (or #buildUsing:) was proposed specifically for the library distribution scenario rather than the application distribution scenario. It's role is analogous to #main:args. You distribute the library, as you distribute the application, as an object its dependencies (excluding the platform). You produce the application (respectively library) object using #packageUsing: (resp. #packageLibraryUsing:). You then instantiate the application (respectively library) object using #main:args: (resp. #buildUsing:). At that the first stage (dev time) you provided a manifest, a global namespace of the classes you need. At the second stage (load time) you provide a platform that ties this code to the standard libraries that provide access to real world capabilities.
Would it be on the 'target system',
Yes.
making "#build: platform" equivalent to the 'package manager' doing the 'installation step' equivalent [item A4 in section A]?
No.
- If so, how and in which format would the result of the invocation of "#package: manifest" be "shipped" to the 'target platform'?
As noted above, there are any number of formats. At the very core, there is a serialized Newspeak object. For a CounterApp, this would be ~100Kb. But because we ship the platform code in a vfuel, weget to about 1Mb. Once we add the engine (300K WASM) and other resources, things grow further, and we are distributing something like zipped directory (e.g. server.zip on the downloads page). If we wrap that in an Electron app, and wrap that in a MacOS app, we have 70Mb zipped, or 175Mb unzipped, because we now are shipping Chromium as well.
- Is the general intent that the tools (the IDE) invokes this method?
The engine invoked #main:args:, which may invoke #buildUsing: (which may invoke other #buildUsing: methods). The #packageUsing: or #packageLibraryUsing: methods are invoked by the IDE, but might also be invoked by other #packageUsing: or #packageLibraryUsing: methods as they build themselves up.
If so (as we do not yet have an example), could you briefly describe how that would work, for example "after downloading the 'program package', in the IDE on top, we will have a button 'install package'" - well, I am being presumptious :)
No, if you are using the IDE, all you need is the source code. Then you can use options like [run] or [debug] see below to instantiate apps, or you can use [deploy]m to redeploy the apps or libs in different formats or whatever.
A bonus question : )What makes a Newspeak class to show the following buttons: [deploy] [configuration] [run] [debug]? I am making an assumption that:
- [deploy] - shows iff the class has the "#package: manifest" type method?
I
shows iff the class has a class method #packageUsing:. That can be either a factory method or another class method. The method name is a convention indicating the class is an app, so it can be deployed.
- [configuration] - not sure what makes this show, perhaps class name ending with Configuration?
No, as above. The class is an app, and it can be deployed, via different deployment configurations (e.g., Javascript, vfuel etc.).
- [run] and [debug] - shows if the class has "#main: platform args: args" method?
No, as above. An app can be run, or debugged, within the IDE.
- [runTests] - shows if the class has " #packageTestsUsing: manifest" method?
Yes, [run tests] and [show tests} are displyed iff the class has a #packageTestsUsing: class method. Again, a convention indicating that the class is a test configuration.
Can we please correct this and or elaborate?
See above.
Cheers, Gilad
Milan Zimmermann
Jul 8, 2021, 10:43:18 PM7/8/21
to Newspeak Programming Language
Gilad,
Thanks for you patiened and detailed response.
At the moment I just wanted to express I am reading this, it's very helpful, I will likely have some follow up clarification questions, but it may take me a day or a few days to find time to internalize and align it.
I do have several not-so-brief comments at this time. Actually I should not be asking to read them or respond as I am still not clear on what I am doing, and this ended up more writing than thinking...
Except I would appreciate comments the last two - 4) is my current plan for this build and distribution experiment, but likely expressed too vaguely, if even marginally right, and 5) which is related to documentation and attribution.
So here are my notes for now:
1) I should have described my reason for getting into this more clearly: I started my"distribution" thought process after reading your earlier discussion with Phil https://groups.google.com/g/newspeaklanguage/c/kHAIE_i7gTc/m/2KwHVGIFBQAJ regarding distributing libraries by implementing the "#build:" (I think it gravitated to the name "#buildUsing:" in the most recent post responses, am I right?). I thought: OK, I will do the steps ( and document them) of a "mocked up build and distribution process (from one developer's IDE to another developer's IDE)". I thought to achieve that by
- creating a mini-app, perhaps just a copy of the CounterApp, with an added a library dependency (perhaps just an library and a class MyLibClass with a #doNothing: method, and invoke myLibClass doNothing: from the modified CounterApp)
- then implementing all the discussed methods in the appropriate classess: #main:args, #packageUsing, #packageLibraryUsing:, #buildUsing (let me call them "the 4 methods" :) ,
- then run an "export" (a serialization) of the classes in the application and the library from one IDE,
- then "load" the serialized classes to another IDE (just another local server) and deserialize.
- But I realized I am missing context and did not know where to start. And the fact that I sort of understood - at least on the high level - how .vfuel is being packaged for deployment actually hurt that process, because vfuel is not what I was thinking to distribute for the purpose of the experiment.
2) From the first read of your responses, perhaps the most helpful single insight for me is the analogy between #main:args: and #build:Using - that #buildUsing: is the intended to be the "libraries equivalent" of #main:args: for apps. That is definitely helping me over the first hump (but I see more on the horizon)
3) I now feel that the process I described in 1), would be a lot of work, with lots of unknowns, around creating my own serialization methods, adding some buttons to the IDE to serialize and load. Well, I cannot even think clearly about it and am likely wrong :) .. So (continued in item 4)
4) So my current "at this moment", vague, and possibly useless approach is as follows:
- As this started with my internal need to undrestand and document the roles and lifecycle of "the 4 methods" #main:args, #packageUsing, #packageLibraryUsing:, #buildUsing, in the Newspeak build and distribution process - I should not stray and complicate it further. So I think:
- If I accept the "distribution" is in source text files, and as before just between two locally run IDEs with their own servers,
- And if I implement the modified CounterApp with an added - as described in item 1 - library dependency ( just an library and a class MyLibClass with a #doNothing: method, and invoke myLibClass doNothing: from the modified CounterApp)
- And if I implement "the 4 methods", most importantly
- in the MyLibClass
- #packageLibraryUsingManifest: just place MyLibClass on the slot
- #build: just return the myLibClass instance
- in the ModifiedCounterApp
- #packageUsingManifest: get MyLibClass from the manifest, then call MyLibClass #packageLibraryUsingManifest: and put the result on a slot
- #main:args: In addition to what is there, add a call to MyLibClass #build: (creating an instance of myLibClass), then call "myLibClass doNothing" to actually make use of the lib
- Then I think I have gone through the steps of build and distribution, although in vain - because when accepting distributing source, the MyLibClass could just be a regular class instantiated in the ModifiedCoulterApp, without needing the #build: and #packageLibraryUsingManifest: ... etc
5) If I end up achieving this "build and distribution experiment", I plan to document it on my blog, which in the last few months only talks about Newspeak, in one continualy updated document (source always copied to https://github.com/mzimmerm/newspeak-doc). But I find large parts of that "blog", especially the Q&A are slightly modified copies of Gilad's text from this group, and other places. Gilad, while I am always doing my best mentioning attributions, I feel like I am "borrowing" too much. My points are two: I just want to double check if that is OK, or if I should make any changes or removals, and second, if this is of any usefulness, perhaps the correct thing would be to move it somewhere more appropriate like a document under Newspeak github (or whichever would work) and make the attribution more clear. But if the current state is fine with you, I am happy to continue documenting things as I did so far.
Thanks
Milan
Gilad Bracha
Jul 9, 2021, 12:41:43 AM7/9/21
to newspeak...@googlegroups.com
Hi Milan,
What you say seems reasonable. It really isn't that complicated. You certainly don't need to write any serialization code. That's built into the IDE (even if it is a bit flakey/slow right now).
I would suggest you get comfortable with simple app deployment (e.g, write an app of your own, from scratch, with a #packageUsing: factory method and a #main:args: instance method. It can be as simple as hello world.
Then you can try your example with #packageLibraryUsing: is just a fancy refinement to make it easier to deal with distributing third party libs.
--
You received this message because you are subscribed to the Google Groups "Newspeak Programming Language" group.
To unsubscribe from this group and stop receiving emails from it, send an email to newspeaklangua...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/newspeaklanguage/a6ec2253-a918-4f05-afb1-69a6495ce4cen%40googlegroups.com.
--
Cheers, Gilad
Milan Zimmermann
Jul 9, 2021, 3:11:46 AM7/9/21
to Newspeak Programming Language
Thanks Gilad for following up.
I am also starting to feel it is not that complicated; in a way mostly convention methods, just it was hard at first to grog what is the intented "lifecycle", what calls what, in which order, etc. I will definitely finish up this app packaging and deployment exercise, with a library added later, and report and document it; also will ask here if I run into something.
My next step is the UI ..or maybe a literal documentation, so instead of in org mode, writing documentation in Newspeak with code embedded ... but that is for a later week to even start thinking about how that would be done (if reasonably possible atm).
Milan
Reply all
Reply to author
Forward
0 new messages