Go core packages

[Bob Cunningham]

While Go is still young, I'd like to propose a slight package
reorganization to make it very clear which packages are so central to Go
language use that they should be considered to be part of the language,
much like the STL is for C++. This would be independent of whether or
not the packages were developed by the Go development crew.

Such packages would have their source included as part of the Go tree,
and would be documented alongside the Go language.

Toward this end, perhaps "core" packages should have either a name or
directory prefix making this distinction clear. Presently, the lack of
a prefix (import "fmt") indicates a special relationship, but there is
nothing preventing anyone adding new non-core packages in the same manner.

At first, I was thinking of moving core packages to a "go/" directory:
import "go/whatever" But that makes it easy for another package
"whatever" to live somewhere else. I'd prefer the "core" package names
to be "special".

To minimize the risk of confusion, perhaps "core" packages should be
prefixed with "go.": "import "go.whatever"

If needed, the "go." prefix (call it "go-dot") could be recognized by
the parser to reference a "special" read-only directory under $GOROOT.

The process of "adopting" a package into the language (into "go-dot"),
making it a "core" package, could be something like:
1. Review the package, and the need for it to become a "core" package.
2. Add a "go." prefix, and add package documentation to the Go tree.
3. Deprecate the external reference.

The goal is to ensure developers know which package they can rely upon
always being present, even in a minimal (stripped-down) Go installation
(say, for embedded use).

Is this something that should be considered now, or is it completely
unnecessary?

Should I be waiting for Godot?

-BobC

[Peter Bourgon]

Why is it important to differentiate between "core" packages and
"extra" packages?

[Bob Cunningham]

Uh, I thought I hit "Save As Draft", not "Send". I was on a boring
conference call when I wrote it. Still, my thoughts may apply, though
another edit was needed to clean it up.

At some point, Go applications will hopefully not always be statically
linked: The target will need to have certain packages present to support
dynamic linking at load-time. Should the "Go language run-time
platform" be defined so that certain package binaries (libraries) are
guaranteed to be present in a "minimal" Go target installation?

If you look at Java, several minimal target configurations/platforms are
defined. The .Net languages have similar guarantees via the various
configurations/flavors of the CLR. In C, we can generally assume stdio
and friends will be present (on anything larger than an 8-bit target).
C++ generally assumes the STL. What assumptions, if any, should be made
about the Go runtime-environment?

There are two extremes: 1) You get no packages at all (just the bare
language, with or without the few functions defined in the language
itself). 2) You can assume everything in the Go git tree will be
present. I think neither extreme is satisfactory, and some minimal
target run-time definition may be appropriate.

If standard target definitions are needed, it may be useful to consider
them now, while Go is small and new, and plan accordingly.

But the real reason I wrote my note was to get to the pun at the end.

-BobC

[Jason Catena]

> At some point, Go applications will hopefully not always be statically linked

> I think neither extreme is satisfactory, and some minimal

> target run-time definition may be appropriate.

Does moving from static to dynamic linking mean you must choose
between not running at all, DLL and versioned-shared-library
difficulties, or storing every old shared library? If so, I vote for
continued static linking, in which each executable has in itself all
it needs to run.

[Bob Cunningham]

So, if 5 Go apps are running, and each is using the "fmt" and "io"
packages, you want 5 copies of each of those packages to reside in memory?

What about on, say, a multitasking phone with limited memory: Do you
want every one of the 1000 apps you install to have their own copies of
even the most common libraries? A copy each in storage, AND, when run,
a copy each in RAM? I can see it now: A phone with a swap partition!

That's just silly, in the long term. It's fine for a young language
that hasn't yet left its sandbox. But if Go ever expects to live in the
big, real world, it will have to efficiently and effectively resolve
such issues.

I can envision several Go runtime platform configurations that could be
standardized: Minimal (embedded), Multimedia (MP3 player), Network/Web
(appliance), Phone, etc. But initially, I'm primarily concerned about
the "Minimal" configuration.

-BobC

[Mike Ramirez]

On Thursday 18 February 2010 11:59:42 Bob Cunningham wrote:

Just ftr, I agree this needs to be done to seperate core from 3rd party
packages. But for now, I use a subdirectory under
$GOROOT/pkg/$GOOS_$GOARCH/gufy for the binaries and for the sources
$GOROOT/src/pkg/gufy -- with package subdirectories. My makefile file for the
sources has TARG=gufy/libname

and able to use an import "gufy/libname"

There is a note somewhere from rob in the list on what files to edit to add
these to the build process when you run ./all.bash but one update wiped my
changes so I just use an ungodly command to update and build everything and
not touch the Go files. It's setup as an alias so I don't have to worry about
it. (aliases live in ~/.bashrc)

alias update_go='cd /opt/go; hg pull && hg update && cd src; ./all.bash &&
mkdir ../pkg/linux_386/gufy; cd pkg/gufy/settings/; make clean; make install
&& cd ../utils; make clean; make install; cd'

This solution is probably the most hackish and ugly but allows everything to
live together and update nicely.

I hope this helps you for now.

Mike

---
A lady with one of her ears applied
To an open keyhole heard, inside,
Two female gossips in converse free --
The subject engaging them was she.
"I think", said one, "and my husband thinks
That she's a prying, inquisitive minx!"
As soon as no more of it she could hear
The lady, indignant, removed her ear.
"I will not stay," she said with a pout,
"To hear my character lied about!"
-- Gopete Sherany

[Jason Catena]

> So, if 5 Go apps are running, and each is using the "fmt" and "io"
> packages, you want 5 copies of each of those packages to reside in memory?

Yes. Because although this might be inefficient on the day the system
is installed, as the particular machine lives and breathes and
acquires new software over years of use, it's more efficient to
package with the program the minimal set of additional code each
program needs. Optimizing memory usage on day 1 impedes the much more
common case of memory usage after an arbitrary amount of
customization.

[Mike Ramirez]

I think the move includes a go package manager for manging the libraries.
which should help allieviate the pain of resolving these dependancies.

Mike

--
There's no saint like a reformed sinner.

[roger peppe]

i don't think it does. i think you can have both static and dynamic
linking in the same go instance, if that's what you want.

for code sharing the same package name space, i think static linking
makes sense. it ensures that all the packages that have been linked with
are always available, and shared, if used from multiple places.

but i think it wouldn't be too hard to provide a go-style dynamic loading
interface.

the interface might look like this, unashamedly inspired by Limbo:

package dynload
func Load(path string, intf interface{}) (code interface{}, err os.Error)

Load loads the dynamically loadable go executable at path,
expecting it to implement the methods found in intf (which must be an
instance of
an interface, used only for its type). The dynamic linker invokes
main.loaded() inside the newly loaded module, which returns an interface
that if compatible with the type of intf, is returned by Load as a
type compatible
with the type of intf.

The idea is to share packages between dynamically-linked
executables only as far as the interface type passed to Load
requires. If a package is mentioned in that interface, then it will be
shared with the loading module. it checks that the loading module's
instance of the package
is compatible, otherwise the load will fail.

This presupposes a definition of package compatibility based on
supersets of package functionality, but i don't think that's too hard,
and it all happens behind the scenes.

The (static) linker can inspect the type signature of main.loaded to make
sure that it does not inline any calls to a package that is exposed in
its interface.

All packages not in the imported interface live in a separate code & data
space (although the code could potentially be shared if identical)

here's how it might look:

// the loader
package main
import (
"fmt"
"dynload"
)
type Command interface {
Main(argv []string)
}
func main() {
mod, err := dynload.Load("/path/to/somewhere.so", Command(nil))
if mod == nil {
fmt.Print("cannot load: %v\n", err)
}else{
mod := mod.(Command)
mod.Main([]string{"hello", "world'})
}
}

// the loaded code
package main
import (
"fmt"
)
type Command interface {
Main(argv []string)
}
type Main int
func (_ Main) Main(argv []string) {
fmt.Printf("foo %v\n", argv)
}
func loaded() Command {
return Main(0)
}

[roger peppe]

On 18 February 2010 21:52, roger peppe <rogp...@gmail.com> wrote:
> package dynload
> func Load(path string, intf interface{}) (code interface{}, err os.Error)

i realise that this can't work, because the loading package has already
excluded unused functions from its text segment, and unless dynload
is privileged, there's no way for the compiler to know which packages
might be shared. pity.

[Uriel]

On Thu, Feb 18, 2010 at 10:05 PM, Bob Cunningham <Fly...@gmail.com> wrote:
> On 02/18/2010 12:23 PM, Jason Catena wrote:
>>>
>>> At some point, Go applications will hopefully not always be statically
>>> linked
>>>
>>
>>
>>>
>>> I think neither extreme is satisfactory, and some minimal
>>> target run-time definition may be appropriate.
>>>
>>
>> Does moving from static to dynamic linking mean you must choose
>> between not running at all, DLL and versioned-shared-library
>> difficulties, or storing every old shared library?  If so, I vote for
>> continued static linking, in which each executable has in itself all
>> it needs to run.
>>
>>
>
> So, if 5 Go apps are running, and each is using the "fmt" and "io" packages,
> you want 5 copies of each of those packages to reside in memory?

This was not an issue on a PDP-11, or Unix in general until Sun
decided that their software was so bloated that static linking
wouldn't do anymore.

And even then the alleged memory and space benefits of dynamic linking
are very questionable, and they come at considerable performance and
complexity cost: http://harmful.cat-v.org/software/dynamic-linking/

> What about on, say, a multitasking phone with limited memory: Do you want
> every one of the 1000 apps you install to have their own copies of even the
> most common libraries?  A copy each in storage, AND, when run, a copy each
> in RAM?  I can see it now: A phone with a swap partition!
>
> That's just silly, in the long term.  It's fine for a young language that
> hasn't yet left its sandbox. But if Go ever expects to live in the big, real
> world, it will have to efficiently and effectively resolve such issues.

Just because many people have been duped by crappy software into
believing certain issues exist doesn't mean they are real.

> I can envision several Go runtime platform configurations that could be
> standardized: Minimal (embedded), Multimedia (MP3 player), Network/Web
> (appliance), Phone, etc.  But initially, I'm primarily concerned about the
> "Minimal" configuration.

Such specialized/embedded environments are the ones that would have
the least use for dynamic linking.

uriel

> -BobC
>
>

[Jessta]

On 19 February 2010 08:05, Bob Cunningham <Fly...@gmail.com> wrote:
> What about on, say, a multitasking phone with limited memory: Do you want
> every one of the 1000 apps you install to have their own copies of even the
> most common libraries?  A copy each in storage, AND, when run, a copy each
> in RAM?  I can see it now: A phone with a swap partition!

ummm...is your executable code usually the major memory hog? I find
that extremely hard to believe.
In theory dynamic linking leads to one copy of the library, in
practice versions of libraries are binary incompatible so you need
multiple versions installed. Then you have the multiple libraries that
do similar things,(eg QT and GTK) where you need the entire library
even if the app is only using a small part of it

The statically linked executables don't each contain a copy of the
entire library, they only contain the parts of the library they are
actually using. So if your executables don't use a part of a library
at all then that part never needs to be installed or loaded.

For a language like Go with a smaller user base, static linking is
actually an advantage. Users are only going to be running a handful of
applications written in Go. So they'd rather not have to install a
giant library 90% of which they aren't ever going to use.

I don't run any ruby apps, haskell apps, C# apps or Java apps, because
of the very few apps I would actually like to use that run on those
platforms I can't justify installing the massive dependences required
just to run that one app.

Plus you'd have to decide on standard core packages that can be
expected to be installed on every Go system, which leads to things
like the python standard library where a lot of the packages are
terrible, but you can't really use anything else because it's probably
not going to be available.

- jessta

--
=====================
http://jessta.id.au

[David Roundy]

On Sat, Feb 20, 2010 at 7:52 AM, Jessta <jes...@gmail.com> wrote:
> I don't run any ruby apps, haskell apps, C# apps or Java apps, because
> of the very few apps I would actually like to use that run on those
> platforms I can't justify installing the massive dependences required
> just to run that one app.

To nitpick: this doesn't apply to Haskell apps, which are statically
linked, just like go, and the binaries have no dependencies apart from
any dynamically-linked C libraries they use, also just like go.

David

[Hans Stimer]

I would like to argue for dynamic linking from the perspective of dynamic program enhancement. The first requirement:

R1: Extend a program's behavior at runtime with code that runs at the speed of statically bound code

1) Some programs need to not terminate but yet still extend behavior

2) The in-progress go interpreter (go eval) will provide for dynamic extension, but will run slower (much slower?) than statically compiled code

3) Dynamic linking can provide nearly the  speed of static linked code

Note, the above isn't an argument for having all libraries be dynamic, just for the ability to choose which libraries are dynamic.

Erlang takes this further and adds the requiement:

R2: Dynamically replace existing some functionality with new functionality.

It does this by allowing 2 versions of the same code to exist simultaneously with a bunch of restrictions. I haven't used Erlang for any serious work because it is slow, but I do see the use for the feature. R2 isn't a requirement for all functionality being hot-swappable.

R1 I believe is harder to work around in the client case. Imagine a game client that wants to add new game logic but not require the user to quit the game to get that logic. Many games use an interpreted language like Lua for this, but at a tremendous performance sacrifice.

For servers, the lack of R1 and R2 has driven architecture. The standard design has been small discrete transactions with short connection times so a new server with the added/modified functionality can be slid in. This requires the state to be kept external from the server i.e. separate database. This works less well if the server contains state internally i.e. in-memory database.

I've created servers (billions of daily requests) that use some kind of interpreted language for supporting R1 and R2, but because of performance it had to be used sparingly. 

All the systems I have designed had to use an external database so that new performance-critical functionality could be added. Had I the ability dynamically update code (R1 and R2) I would of designed a distributed in-memory state system with replicated state for reliability. I would probably still use a disk based transaction log in case of catastrophe. 

For completeness I'll mention the space saving requirement:

R3: Save memory and disk space when multiple executables are using the same libraries

Personally, I'm somewhat in favor of R3, but I have to admit that I see less need for it than R1 and R2.

If we end up having dynamic libraries then I hope we can agree to:

R4: Dynamic extensions can be selected by name and version

R5: Dynamic extensions can be validated for authorship and integrity

[Ivan Wong]

It seems like one important feature of dynamic linking wasn't
raised: bug fix upgrades of libraries. What about if libfoo is used
by like 500 binaries on your system, and there's a security bug fix
upgrade of libfoo? You have to recompile all 500 binaries, not
mention that you have to find the list of the binaries needed to be
re-compiled. Unless you say you don't care about security problems
or bugs you have to pay attention to this problem and this can
really kill people.

When you say libraries are incompatible across versions, this is not
always true for all libraries. I am not talking about some libraries
which are developed in-house for a company where you can find no
development discipline. Many serious library development follows
strictly some versioning systems (e.g.,
http://sources.redhat.com/autobook/autobook/autobook_91.html#SEC91),
so distros like debian can have bug fix upgrades of a library like
libgtk+ which breaking all applications that depend on it.

-ivan

--
Ivan Wong <iva...@gmail.com>
GPG: 1024D/40510AB7: 88BF A832 50D7 30F0 3850 DE17 049D A727 4051 0AB7

[DaanS]

I'd like to take a step back, and separate the "standard packages" and
"dynamic linking" issues.

Once Go leaves the sandbox stage (and probably even before then)
people are going to want to create alternative implementations.
Keeping these implementations compatible is hard enough if you have a
standard (see cross-browser html/css/javascript, or GNU vs. Microsoft
C/C++), and next to impossible without one. So I'd say that a well-
defined list of standard interfaces (read: packages) is pretty much
mandatory if Go is to become a mainstream production language.

But I'm talking about a language standard here. Something that tells
programmers that they can expect a function fmt.Printf to be present,
and to act in a certain manner. Things like static vs. dynamic linking
would be way beyond the scope of such a document.

Linking isn't an aspect of languages, but of toolchains and runtimes.
And before anyone even thinks of arguing static vs. dynamic, or
whatnot, we should first decide whether we even want a standardized
toolchain and/or runtime. If you look at Java or C# marketing material
you might think it's pretty much a given. But C and C++ have been
around for decades, and they seem to do just fine without.

Daan

[Russ Cox]

> But I'm talking about a language standard here. Something that tells
> programmers that they can expect a function fmt.Printf to be present,
> and to act in a certain manner.

It seems to me premature to lock down what's in the
standard packages, given how rapidly both the language
and the packages are changing. For example, fmt.Printf
today has a different signature than it did at the beginning
of February. The helper functions strings.Bytes and strings.Runes,
both of which would have certainly been "core" functionality
two weeks ago, today do not exist.

We're just not at a point where standardization makes sense.
If for some reason you absolutely must have a package
standard today, use the contents of src/pkg as your standard.
But it will be different tomorrow.

Russ

[Bob Cunningham]

That nicely addresses my original question that started this thread. If
I may loosely summarize and paraphrase:

Much in Go-land is or may be in flux. Application longevity and
development consistency still are small issues compared to getting the
language and its packages "right". We aren't there yet, and it is
difficult even to tell how close we are: This could go on for many more
months, possibly years.

To borrow a nautical metaphor:

We Go-nuts are on a shared voyage, and while the Go Crew collectively
are responsible for maintaining and upgrading the ship, they aren't the
only ones giving steering input. About all we really know is where we
started (a new systems programming language with CSP and interfaces),
and it is way too soon to see what the ship itself or the final
destination will look like.

At best, all we "passengers" can really do is help paddle (write Go
code), and shout when our paddle hits something unexpected. Be prepared
to paddle in a circle once in a while.

It is also OK if you only want to lay back on the sun deck and see how
the voyage goes. It's a great view!

-BobC

[Jessta]

On 9 March 2010 09:22, Bob Cunningham <Fly...@gmail.com> wrote:
> It is also OK if you only want to lay back on the sun deck and see how the
> voyage goes.  It's a great view!
>

Thanks, that was a rather relaxing thing to read with my morning coffee.

[DaanS]

> > But I'm talking about a language standard here. Something that tells
> > programmers that they can expect a function fmt.Printf to be present,
> > and to act in a certain manner.
>
> It seems to me premature to lock down what's in the
> standard packages, given how rapidly both the language
> and the packages are changing.

Absolutely. What I meant to say was that Go will need a language
standard before it leaves the sandbox stage, and not that it needs one
today. Sorry if I didn't make that clear.

[Russ Cox]

> Absolutely. What I meant to say was that Go will need a language
> standard before it leaves the sandbox stage, and not that it needs one
> today. Sorry if I didn't make that clear.

Does that mean Perl, Python, Ruby, Scala, and Erlang are
all still in the sandbox stage?

Russ

[Eleanor McHugh]

Speaking just for Ruby, there is both a Core Library and a Standard Library. There's also a proposed ISO standard (though I'm far from convinced that's a good idea) and the RubySpec test suite which is becoming the de facto standard.

Go doesn't necessarily need any of these things to be an effective language, but for casual users having at least a core library will certainly help.

Ellie

Eleanor McHugh
Games With Brains
http://feyeleanor.tel
----
raise ArgumentError unless @reality.responds_to? :reason

[DaanS]

Since I don't follow the development of those languages, I don't
really feel qualified to make that judgment myself. But at a glance,
all of those languages seem to have defined and documented a set of
"core" packages/modules/etc.. If their definitions are reasonably
stable, then I'd say that (at least in this area) they've left the
sandbox stage.

Daan