Implementation roadmap, very very good
Alex Combas
Implementation roadmap
- Improved garbage collector, most likely a reference counting collector with a cycle detector running in a separate core.
- Debugger. <- Good!
- Native Client (NaCl) support. <- Super extra plus Good!
- App Engine support. <- Super Good!
- Improved CGO including some mechanism for calling back from C to Go.
- SWIG support.
- Public continuous build and benchmark infrastructure.
- Improved implementation documentation.
- Package manager, possibly including a language change to the import statement.
Jessta
> Implementation roadmap
> Package manager, possibly including a language change to the import
> statement.
I hope this language change involves urls of some kind.
Compile time package fetching(with caching) would be nice. Something I
always find annoying about old code that uses third-party libraries is
that 5 years later you don't have the third party libraries and nobody
has documented where you can get it from.
--
=====================
http://jessta.id.au
baldmountain
On Dec 18, 1:29 am, Jessta <jes...@gmail.com> wrote:
> 2009/12/18 Alex Combas <alex.com...@gmail.com>:
>
> > Implementation roadmap
> > Package manager, possibly including a language change to the import
> > statement.
>
> I hope this language change involves urls of some kind.
> Compile time package fetching(with caching) would be nice. Something I
> always find annoying about old code that uses third-party libraries is
> that 5 years later you don't have the third party libraries and nobody
> has documented where you can get it from.
>
I'd like to see a golang.org endorsed package management system.
Something like RubyGems. Maybe hosted at code.google.com? And it
doesn't have to have compile time code fetching, although it would be
nice. Even if its just a hint about what is missing and community
knowledge about where to start looking for it. ie. gem install foo
One of the things that is going to make this difficult is that a good
number of Go libraries are going to be wrappers around existing C/C++/
other libraries. These secondary dependencies are going to be a pain
to track. All this is going to require some thought and design. I hope
someone is thinking about it.
geoff
John Asmuth
some sort of http based importing.
On Dec 18, 1:29 am, Jessta <jes...@gmail.com> wrote:
> 2009/12/18 Alex Combas <alex.com...@gmail.com>:
>
Devon H. O'Dell
NoiseEHC
Implementation roadmap
- Improved garbage collector, most likely a reference counting collector with a cycle detector running in a separate core.
Ian Lance Taylor
>> Implementation roadmap
>>
>> * Improved garbage collector, most likely a reference counting
We think that a reference counting approach will take better advantage
of multicore and permit the program to run without pausing. This
remains an area of investigation.
Ian
Esko Luontola
alternatives have you evaluated? (I'm working on an application server/
object database where I'll need a garbage collector, and am also
thinking that reference counting would be the best to achieve low
latency.)
Qtvali
Objective-C has very fast one with reference counting plus some kind
of automated reference counters (similar to defer, one can delay
release):
http://developer.apple.com/mac/library/documentation/cocoa/Conceptual/MemoryMgmt/MemoryMgmt.html
This is actually very robust and simple thing combining few rules and
simple library - one could implement some smaller similar thing in C++
probably in one evening.
I think that implementing algorithm what Go is going to use would need
more effort (just by the fact that Go-s garbage collector does).
Qtvali
some kind of destructor method is certainly needed, be it a standard
or built-in.
On Dec 19, 7:19 pm, Qtvali <qtv...@gmail.com> wrote:
> On Dec 19, 1:28 pm, Esko Luontola <esko.luont...@gmail.com> wrote:
> > Which cycle collector algorithm you are considering to use? Which
> > alternatives have you evaluated? (I'm working on an application server/
> > object database where I'll need a garbage collector, and am also
> > thinking that reference counting would be the best to achieve low
> > latency.)
I give you a basics how Objective-C garbage collector works:
* Each object has int field with reference count
* Programmer explicitly calls "retain" and "release", which will
increment or decrement this reference count
* Objects must have hierarchical "ownership" tree for classes with
circular references
* There is an object called "retain pool", which internally contains
an array of objects, which "release" method it has to call. There is
one active retain pool for each thread - when you create a new pool,
it becomes active and when you release it, it will send enqueued
release messages to all registered objects. When you create an object
and want it to get release message when you deallocate current retain
pool, you call "autorelease" instead of "release" on that object.
** Usually you create and release retain pool in same method, which is
calling submethods. For example, you might do it in main loop of a
thread and/or in each iteration of this loop; also in any method,
which is going to allocate big amount of temporary objects.
This kind of garbage collector is simple to implement - you must add
the retain count field to objects you want to manage; then create an
autorelease pool containing dynamic array of object pointers and
retain count of itself; then you must implement a method to decrease
retain count, which eventually calls destructor; then a destructor of
retain count, which calls this method for all pointers in it's array;
then an autorelease method, which you might implement as pool's method
first for simplicity.
This kind of GC is very fast - it won't do much for yourself, but it
does exactly as much as needed to keep memory management relatively
simple when related to C and C++, it doesn't also do the stop-the-
world kind of things, which Java does when things get out of control.
John Asmuth
context of an event driven callback program.
The way it works is the "main" function is essentially hidden from
you, and goes through cycles of polling events from input sources,
sending them to the relevant listeners, and then deallocating as
necessary.
It's not so much a part of obj-c, but a part of the memory manager
apple made for the framework they provide for GUI programming (the
retain, release, autorelease etc come from the NSObject class).
As such, it doesn't directly apply to the go context. In obj-c, you
have access to primitive alloc and dealloc methods (or messages),
which the GUI framework makes use of with its reference counting. In
go-land, there is no such thing, and no event loop built into the
language to allow this to be done behind the scenes.
- John
On Dec 19, 12:19 pm, Qtvali <qtv...@gmail.com> wrote:
> On Dec 19, 1:28 pm, Esko Luontola <esko.luont...@gmail.com> wrote:
>
> > Which cycle collector algorithm you are considering to use? Which
> > alternatives have you evaluated? (I'm working on an application server/
> > object database where I'll need a garbage collector, and am also
> > thinking that reference counting would be the best to achieve low
> > latency.)
>
> Objective-C has very fast one with reference counting plus some kind
> of automated reference counters (similar to defer, one can delay
> release):http://developer.apple.com/mac/library/documentation/cocoa/Conceptual...
Ian Lance Taylor
Qtvali
> The obj-C memory manager you are referring to only makes sense in the
> have access to primitive alloc and dealloc methods (or messages),
> which the GUI framework makes use of with its reference counting. In
> go-land, there is no such thing, and no event loop built into the
> language to allow this to be done behind the scenes.
>
> - John
Yes.
My answer was not directly related to Go - I have nothing to say about
Go's garbage collecting.
Esko Luontola asked here, which garbage collector Go is using,
mentioning that he needs one for his application server/object
database, which should have low latency. Application servers often
have some kind of structure you described, thus it might be that this
Obj-C kind of memory management would be actually better to Esko than
anything, what will be implemented in Go - given that it is simple
enough to be implemented as part of another project, which is not a
programming language. If you are creating application server in a
language, which does not have it's own GC, it's relatively hard to
implement one, which will be natively supported - I think that trying
to do it in the same way with Go would blur the goal to get this
application server ready some day. It would be much simpler to port it
to Go ;)
Kevin Ballard
Tambet
Tambet
Kevin Ballard
Tambet
--
Tambet
Mike Zraly
Well, it could be used to implement automatic reference counting, but that's not the same thing as garbage collection. Besides the difference in performance characteristics, reference counting has one significant flaw that garbage collection doesn't, which is the inability to deal with retain cycles. This is especially a problem in an automatic reference counting situation, as the programmer does not have an easy way to detect these retain cycles and break them, besides manually identifying when two objects reference each other and manually nilling out the fields. Using reference counting in a language like Go has another big flaw, which is that it requires adding an invisible retain count field to every single struct, which could significantly bloat the memory usage of programs that use lots of small structs. For example, a simple linked list would normally be implemented as a struct that looks something liketype LinkedNode struct {prev *LinkedNode;next *LinkedNode;value interface{};}Adding a retain count field to this struct would increase the memory footprint by 33%, and in this case it would also introduce a retain cycle for every single pair of nodes, requiring the programmer to manually destroy the linked list when he is done with it and completely eliminating any and all gains from automatic memory management.Garbage collectors were created for a reason. I suggest we stick with using them rather than going to an automatic reference counting system.
I don't think anyone is suggesting go give up automatic garbage collection. But don't discount advanced reference counting algorithms as a reasonable implementation technique -- may of the objections you raised have been addressed over the years:
- Reference counting garbage collectors are normally paired with cycle detectors or tracing collectors that collect the garbage missed by pure reference counting. In some instances these alternate collectors run concurrently with the rest of the program, so there is no need to "stop the world".
- Reference counts may be stored separately from the objects they hold counts for, and counts may be packed into just a few bits each. It is rare for reference counts to get that large, so some implementations save space by using small counters, never decrementing counters that have reached their maximum, and relying on concurrent cycle collectors or mark-sweep algorithms to collect objects whose counters have maxed out.
- Reference count updates (both increments and decrements) may be queued rather than applied immediately. Some updates may go away altogether, especially for pointers that are updated repeatedly. Consider a stretch of time in which some pointer P gets set to a series of values T1, T2, ..., Tn. Within that time each increment for T2 ... Tn-1 is countered with a decrement, so a lazy reference counter may avoid updating the count for the object altogether. This is called "sliding views".
For a very quick overview of what can be done with reference counting, see here:
http://www.cs.technion.ac.il/~erez/presentations/lp-seminar.ppt
Esko Luontola
> Esko Luontola <esko.luont...@gmail.com> writes:
> > Which cycle collector algorithm you are considering to use? Which
> > alternatives have you evaluated? (I'm working on an application server/
> > object database where I'll need a garbage collector, and am also
> > thinking that reference counting would be the best to achieve low
> > latency.)
>
>
> Ian
Thanks. I've already been looking at the
http://www.research.ibm.com/people/d/dfb/papers/Bacon01Concurrent.pdf
paper, so is that the same algorithm you are going to use in Go? I'm
also looking at http://www.research.ibm.com/people/d/dfb/papers/Paz05Efficient.pdf
which should be an improvement over the previous paper.
⚛
> Implementation roadmap
>
> - Improved garbage collector, most likely a reference counting collector
> with a cycle detector running in a separate core.
> - Native Client (NaCl) support*. <- Super extra plus Good!*
> - App Engine support. *<- Super Good!*
> - Improved CGO including some mechanism for calling back from C to Go.
> - SWIG support.
> - Public continuous build and benchmark infrastructure.
> - Improved implementation documentation.
> - Package manager, possibly including a language change to the import
> statement.
Maybe I am the only one, but I am missing the following performance-
oriented feature:
- compile-time evaluation, with an ability to use [compile-time
generated objects] at run-time
I think this would make a difference in application startup time, for
example.
befelemepeseveze
> - compile-time evaluation, with an ability to use [compile-time
> generated objects] at run-time
Sounds like Go literals to me.
Helmar
On Dec 28, 11:44 am, befelemepeseveze <befelemepesev...@gmail.com>
wrote:
Well, but calculated literals. Maybe by a func() or so, maybe using
arbitrary packages, maybe the init process is an own, very complex,
program.
Regards,
-Helmar
PS: @Alex: you are not alone.
⚛
wrote:
As far as I know, package-level variables cannot be evaluated at
compile-time in Go. They are evaluated when the package is first
accessed or loaded. This means they are evaluated at run-time. Or am I
mistaken?
Ian Lance Taylor
Conceptually package-level variables are initialized at runtime. But
if a package-level variable is initialized to a completely constant
value, then that initialization should happen at compile time.
Ian
befelemepeseveze
> As far as I know, package-level variables cannot be evaluated at
> compile-time in Go. They are evaluated when the package is first
> accessed or loaded. This means they are evaluated at run-time. Or am I
> mistaken?
You're right AFAIK. The compiler will evaluate what it evaluate can.
That doesn't include e.g. memory addresses returned by the allocator
only at runtime. And IIRC a literal as part of an expression gets
always a fresh alloc.
⚛
> ⚛ <0xe2.0x9a.0...@gmail.com> writes:
> > As far as I know, package-level variables cannot be evaluated at
> > compile-time in Go. They are evaluated when the package is first
> > accessed or loaded. This means they are evaluated at run-time. Or am I
> > mistaken?
>
> Conceptually package-level variables are initialized at runtime. But
> if a package-level variable is initialized to a completely constant
> value, then that initialization should happen at compile time.
>
> Ian
For example: Let's have a constant list of strings assigned to a
package-level variable. Suppose I want to sort those strings in order
to be able to *later* perform a search in O(log N) time. Maintaining
the sorted list is (in some/most cases) tedious, an unnecessary burden
to the programmer. The most rational course of action seems to be to
perform the sorting at compile-time. This can save (in certain cases)
precious milliseconds of CPU time which would be otherwise spent on
doing the sorting at run-time. The benefit might be that (certain)
applications written in Go will be able to start like "Zap! There it
is".
A problem might be that objects generated at compile-time might be big
in terms of how many bytes they consume in the final executable/
library. Therefore, the programmer should be allowed to *easily*
determine the size of generated compile-time objects.
befelemepeseveze
> For example: Let's have a constant list of strings assigned to a
> package-level variable. Suppose I want to sort those strings in order
> to be able to *later* perform a search in O(log N) time. Maintaining
> the sorted list is (in some/most cases) tedious, an unnecessary burden
> to the programmer. The most rational course of action seems to be to
> perform the sorting at compile-time.
Compiler executing some of the just compiled code really doesn't look
rational to me. Instead of compile time, such sort maintenance code
should be IMO executed within make by a invoking a specific code
generator utility with the proper dependency on the string list data
file.
⚛
wrote:
> On 29 pro, 11:25, ⚛ <0xe2.0x9a.0...@gmail.com> wrote:
>
> > For example: Let's have a constant list of strings assigned to a
> > package-level variable. Suppose I want to sort those strings in order
> > to be able to *later* perform a search in O(log N) time. Maintaining
> > the sorted list is (in some/most cases) tedious, an unnecessary burden
> > to the programmer. The most rational course of action seems to be to
> > perform the sorting at compile-time.
>
> Compiler executing some of the just compiled code really doesn't look
> rational to me.
It can be done safely in a separate process.
> Instead of compile time, such sort maintenance code
> should be IMO executed within make by a invoking a specific code
> generator utility with the proper dependency on the string list data
> file.
What if it's not a simple list of strings. Are you going to write a
specialized wrapper for each data type to read it from the external
file?
befelemepeseveze
> It can be done safely in a separate process.
And how it's then significantly different from the make on demand
invoked maintainer/generator except for not compiling/invoking that
generator every time?
> What if it's not a simple list of strings. Are you going to write a
> specialized wrapper for each data type to read it from the external
> file?
No as that would be probably the most expensive way how to do it.
Ben Tilly
[...]
> For example: Let's have a constant list of strings assigned to a
> package-level variable. Suppose I want to sort those strings in order
> to be able to *later* perform a search in O(log N) time. Maintaining
> the sorted list is (in some/most cases) tedious, an unnecessary burden
> to the programmer. The most rational course of action seems to be to
> perform the sorting at compile-time. This can save (in certain cases)
> precious milliseconds of CPU time which would be otherwise spent on
> doing the sorting at run-time. The benefit might be that (certain)
> applications written in Go will be able to start like "Zap! There it
> is".
You don't seem to understand that having good compile time performance
is an explicit design goal in Go. Moving arbitrary computations into
compile time is completely counter to this design goal. I'm not
concerned here about the code that I write, but rather about the
external libraries I load whose "clever" authors moved work into
compilation. As I load more and more of those I will wind up with
bloated executables and slow compilation time.
An alternate design strategy for your goal is to memoize the data
structure to an external data file at runtime in an initialization
step. This requires no language change, keeps compilation fast, and
moves the performance penalty to the first time you run the program.
Furthermore putting the data structure in its own file makes it easy
to see how much space it costs.
To me the fact that this requires more effort and thought is a bonus
because it limits how much people will slow down compilation because
of premature optimization.
Cheers,
Ben
Qtvali
> > For example: Let's have a constant list of strings assigned to a
> > package-level variable. Suppose I want to sort those strings in order
> > to be able to *later* perform a search in O(log N) time. Maintaining
> > the sorted list is (in some/most cases) tedious, an unnecessary burden
> > to the programmer. The most rational course of action seems to be to
> > perform the sorting at compile-time. This can save (in certain cases)
> > precious milliseconds of CPU time which would be otherwise spent on
> > doing the sorting at run-time. The benefit might be that (certain)
> > applications written in Go will be able to start like "Zap! There it
> > is".
First, Go currently does not have immutable lists.
Second - if user wants this list in sorted manner, it is worth to have
sorted also in code. If an order is important property of this list,
then I don't understand, why you want to show it unsorted to other
programmers. Manually sorting short list is not much overhead (given
that you must first create this list). For long lists, if there is any
reason to have them sorted in code, I suggest you to use external
tools to manage this list - why to create this time overhead in
compilation process if you can already have it sorted when starting
compilation? Also, it's hard to spot accidental double-entries from
unsorted lists, which is why I personally prefer to sort some things
even if this is not at all important to user to have them sorted. If
some code contains unsorted list, even if user never sees the order, I
would consider it sloppy code in most cases.
⚛
> > On Tue, Dec 29, 2009 at 2:25 AM, ⚛ <0xe2.0x9a.0...@gmail.com> wrote:
> > > For example: Let's have a constant list of strings assigned to a
> > > package-level variable. Suppose I want to sort those strings in order
> > > to be able to *later* perform a search in O(log N) time. Maintaining
> > > the sorted list is (in some/most cases) tedious, an unnecessary burden
> > > to the programmer. The most rational course of action seems to be to
> > > perform the sorting at compile-time. This can save (in certain cases)
> > > precious milliseconds of CPU time which would be otherwise spent on
> > > doing the sorting at run-time. The benefit might be that (certain)
> > > applications written in Go will be able to start like "Zap! There it
> > > is".
>
> First, Go currently does not have immutable lists.
Immutability can be achieved by combining encapsulation and visibility
restrictions. Go has both. So, it's not impossible to implement
immutable lists in Go, although it will not look as "nice" as in a
language supporting transitively constant objects.
Besides, the term "constant object" is artificial, in the sense that
*any* constant object is constant only on a certain abstraction level
or within certain limits. (I don't know if you understand what I mean
by this.)
> Second - if user wants this list in sorted manner, it is worth to have
> sorted also in code.
Which implies that there exists someone or something which is
responsible for maintaining the sorting (including checking for
duplicated entries, if that is a requirement).
In your case, this responsibility is *solely* on the programmer who
needs to (because it is a piece of Go code) be maintaining the
ordering.
In my case, the responsibility is *either* on the programmer *or* on a
piece of code created by the programmer. It gives the programmer the
ability to automate the maintenance&checking work fully *within* the
capabilities of the programming language. Whereas, in your approach,
if the programmer wants to automate it then it has to happen *outside*
of the language (e.g: an external utility invoked from a Makefile).
Note that my approach/suggestion does *not* force the programmer to
use it, it just provides an option which the programmer can use.
> If an order is important property of this list,
> then I don't understand, why you want to show it unsorted to other
> programmers.
You are implying stuff. I never said I want to show it to other
programmers, packages or source-code files. An intelligent programmer
will use [information hiding built in to the programming language] to
his/her advantage - I have no doubts about that. As for stupid
programmers, I absolutely don't care what they will do.
> Manually sorting short list is not much overhead (given
> that you must first create this list). For long lists, if there is any
> reason to have them sorted in code, I suggest you to use external
> tools to manage this list - why to create this time overhead in
> compilation process if you can already have it sorted when starting
> compilation?
I think you have forgotten that the Go file in question needs to be
recompiled only when it changes or its dependencies change. It is not
like you are recompiling it every time you run "make". I highly doubt
that real-world overhead would be greater than using an external tool
invoked from a Makefile.
> Also, it's hard to spot accidental double-entries from
> unsorted lists,
You could check for duplication during the compile-time execution. As
a matter of fact, you could program *any* check you want.
Qtvali
You want to put some huge databases into code?
I think, anyway, that for such case it's nice to create some
generator. You can include this generator in your makefile. You can
make it check change dates.
⚛
> On Dec 30, 1:52 pm, ⚛ <0xe2.0x9a.0...@gmail.com> wrote:
>
> > On Dec 29, 7:39 pm, Qtvali <qtv...@gmail.com> wrote:
> > > Also, it's hard to spot accidental double-entries from
> > > unsorted lists,
>
> > You could check for duplication during the compile-time execution. As
> > a matter of fact, you could program *any* check you want.
>
> You want to put some huge databases into code?
I don't understand why are you asking this. It was a rhetoric
question, I guess.
The decision/responsibility whether it is better [to put the static
data into a Go source file] or [rather put it into an external file]
is on the shoulders of the programmer.
> I think, anyway, that for such case it's nice to create some
> generator. You can include this generator in your makefile. You can
> make it check change dates.
Let me rephrase what you just wrote: you have no problems with an
external generator implemented in a (perhaps) different language, but
you have problems with the idea of having the generator implemented
directly in Go. Is that what you mean?
befelemepeseveze
> The decision/responsibility whether it is better [to put the static
> data into a Go source file] or [rather put it into an external file]
> is on the shoulders of the programmer.
There's no need to separate those two concepts. The generator can
quite easily (Go and it's library packages already offers the right
tool set) maintain (e.g. sort) the static data set inside a Go source
file in the form of Go literals. That's why I previously mentioned
that writing a wrapper code for this is unnecessarily expensive.
⚛
wrote:
You are right. It's certainly an interesting way of how to do it.
Question: What if the input data contains some function calls? Those
calls would be executed during the compile-time evaluation.
befelemepeseveze
> Question: What if the input data contains some function calls? Those
> calls would be executed during the compile-time evaluation.
I'm not sure if I understand the problem, so now I'd only guess...
Let there be a generator app, composed of files data.go and
generate.go (both are package main of the generator app [that's
declared with the GOFILES= in it's Makefile]). data.go has some
primary data in the form of Go variable initialization statements
(e.g. v := some literal expression1 \n v2 := ...) and those literals
include expressions which are to be evaluated by some functions. Let's
write those functions to data.go for simplicity as they could also go
to some separate file, let's say eval.go (still package main of the
generator app). In the main() of generate.go you can then simply refer
to the literals of data.go as they are (at runtime of the generator
app) already evaluated - compiler has taken care of this surely before
you can access such v, v2, ... and this is where you get the compile
time evaluation. The main() of generate.go can then just write out the
now *evaluated values* of v, v2, ... to let's say evaluated.go. If
those instances happen to be simple enough types, then the essence of
this write out could be as simple as fmt.Printf("vn := %#v\n", vn) per
variable instance. Prepend the header (package main or something) to
evaluated.go either early in main() of generate.go or using cat for
example or any other way and you are mostly done.
Then your ligthning-fast-starting-App is composed of some let's say
main.go and (the generated) evaluated.go, with the later having a
Makefile dependency on data.go (and possibly eval.go), so if you
update data.go (and/or eval.go), make of App will not forget to
recompile the generator app and execute it (only once, but still you
have to write down this mechanism to the Makefile by hand) to produce
up to date evaluated.go before compiling any part of App which depends
on it == is referring to it in any way.
There could be some issues with the Makefile(s) but that depends on
how the components will be laid out in the directory structure. I
guess the generator should better go to a subdir of App.
Apologizes if I'm talking about something other than what was meant by
the question.
⚛
I think your understanding of the question was OK.
The only point, a significant one, where I do not agree is where you
used "fmt.Printf("vn := %#v\n", vn)". The result of a compile-time
evaluation is in general an object (lets call it "H"). If the result
is a complex structure, the object H assigned to some package-level
variable is merely a handle/root/gate to the whole thing. It would be
more correct to say that a compile-time evaluation produces a web of
objects, and a Go variable(s) gives access to the gate-object(s) only.
However, the web of objects can be packed into a continuous region of
memory and it is possible for this packed data can be saved into a
".o" file which can be linked into a library/executable. So, in my
opinion, serializing the result by means of a custom-coded code like
"fmt.Printf("vn := %#v\n", vn)" is unfortunate, since there exists a
universal way of how to store *any* kind of a result directly into the
final library/executable as binary data.
And there is also an additional benefit of doing it the way I just
described: If the result is read-only, then it is potentially possible
for the OS to map the binary data into the address-space of the
process via read-only pages. This helps to keep memory consumption
down in case the OS runs multiple instances of the same Go
application. On the other hand, should the evaluation happen at run-
time at the start of an application, the page-sharing across multiple
processes is not possible. But I agree there are also certain
problems: (1) if Go chooses to use reference-counting for doing
garbage collection, the page sharing will probably not happen, and (2)
the page-sharing will not happen if the binary data needs to be
relocatable to another virtual address.
Both (1) and (2) are optional, I wonder which way will Go decide to
go.
... decisions made by a baby language can profoundly affect its future
development.
befelemepeseveze
> The only point, a significant one, where I do not agree is where you
> used "fmt.Printf("vn := %#v\n", vn)". The result of a compile-time
> evaluation is in general an object (lets call it "H"). If the result
> is a complex structure, the object H assigned to some package-level
> variable is merely a handle/root/gate to the whole thing.
That's why was previously explicitly noted this:
[[ If those instances happen to be *simple enough types*, then the
essence of this write out could be as simple as fmt.Printf("vn := %#v
\n", vn) per variable instance. ]]
General object doesn't happen to be *simple enough type* for the "%#v"
thus your note
doesn't apply to what was written here.
> However, the web of objects can be packed into a continuous region of
> memory and it is possible for this packed data can be saved into a
> ".o" file which can be linked into a library/executable.
Sure. That was not the topic and couldn't be. This is about
executable's binary
resources which AFAIK Go doesn't support (yet?) on the language level.
The post was about handling the case with the existing only features
of Go.
Still, I also guess, that it may be possible (as you suggest) to link
some data only .o and refer to it with a few only lines of C while
using CGO. If you are going to try this and if you gonna succeed, it
would be nice to post about your experiences to this list - it will be
useful for someone else for sure.
Best regards,
bflm
⚛
On Jan 4, 10:17 am, befelemepeseveze <befelemepesev...@gmail.com>
wrote:
> On 3 led, 18:31, ⚛ <0xe2.0x9a.0...@gmail.com> wrote:
>
> > The only point, a significant one, where I do not agree is where you
> > used "fmt.Printf("vn := %#v\n", vn)". The result of a compile-time
> > evaluation is in general an object (lets call it "H"). If the result
> > is a complex structure, the object H assigned to some package-level
> > variable is merely a handle/root/gate to the whole thing.
>
> That's why was previously explicitly noted this:
>
> [[ If those instances happen to be *simple enough types*, then the
> essence of this write out could be as simple as fmt.Printf("vn := %#v
> \n", vn) per variable instance. ]]
>
> General object doesn't happen to be *simple enough type* for the "%#v"
> thus your note
> doesn't apply to what was written here.
No problem. But my point was that there exists a universal way of
writing any object type. In light of this point, it seems irrational
to use fmt.Printf, or to segregate objects into "simple" and "complex"
ones. In other words, you can use Printf for simple-enough types - I
have no problems accepting that - but I think it's pointless if you do
so.
> > However, the web of objects can be packed into a continuous region of
> > memory and it is possible for this packed data can be saved into a
> > ".o" file which can be linked into a library/executable.
>
> Sure. That was not the topic and couldn't be. This is about
> executable's binary
> resources which AFAIK Go doesn't support (yet?) on the language level.
> The post was about handling the case with the existing only features
> of Go.
OK.
> Still, I also guess, that it may be possible (as you suggest) to link
> some data only .o and refer to it with a few only lines of C while
> using CGO. If you are going to try this and if you gonna succeed, it
> would be nice to post about your experiences to this list - it will be
> useful for someone else for sure.
Hmm, I have my own programming language to which I am trying to add
certain features. I do *not* think I am going to port any of them into
Go. Right now, I am implementing some kind of compile-time evaluation
allowing a method to have code which will be evaluated at the call-
site. But this is only required because I want to enable a certain
feature in the type-system of the language (so, it's a type-system
related thing - whereas my suggestion to add compile-time evaluation
to Go isn't related to the type-system of the Go language).
On another note, if I were to use Go to implement some project of
mine, then I would like to have compile-time evaluation built into the
Go language. Well, to tell the truth, the most needed "feature" I
think Go needs is an IDE. To have compile-time evaluation is of less
importance than to have a moderately good IDE.
Paolo 'Blaisorblade' Giarrusso
existing languages.
Compile-time evaluation is achieved for instance through C++ template
metaprogramming. They also allow to generate code, and well, maybe you
want to separate code generation and compile-time evaluation.
If you are using C++ it's a rather useful feature, but that's a huge
"if" in this context. The current reality shows two facts:
1) The syntax in C++ it's horrible (because this feature was not
designed), but that was easily remedied by the D programming language
(a C++ successor)
2) You can write libraries so that the client code is partially
executed at compile-time. For instance, Boost::Spirit is similar to
parser combinator libraries of most non-lisp functional languages,
except that parsers are composed together at compile-time (to allow
inlining of their code). Or Boost::MPL builds data and control
structures to ease also compile-time evaluation. Well, C++ compilation
times when using those libraries tend to increase enormously, and
that's against Go's aim. Could this be avoided? Basically compilers
here behave as huge AST-based interpreters. Fixing this requires
compiling a piece of user code, executing it during compilation, and
storing the result in the binary.
After all it may be simpler to just arrange for this in the Makefile,
almost like suggested. I'd just add that a serialization mechanism,
based on the existing (or augmented) reflection facilities.
⚛
package main
import (
"fmt"
)
func main() {
fmt.Printf("Hi!\n");
}
I compiled this program with multiple versions of Go. The versions I
will be talking about here are from the following dates: 2010.feb.09
and 2010.apr.19
I then run the binary like this:
perf stat -- ./hi-2010.02.19
Now, the numbers:
$ perf stat -- ./hi-2010.02.19
Performance counter stats for './hi-2010.02.19':
1.675115 task-clock-msecs # 0.878 CPUs
2761579 cycles # 1648.591 M/sec
2477098 instructions # 0.897 IPC
$ perf stat -- ./hi-2010.04.28
Performance counter stats for './hi-2010.04.28':
5.987851 task-clock-msecs # 0.963 CPUs
7277808 cycles # 1215.429 M/sec
10729722 instructions # 1.474 IPC
Notice that the 1st binary executes 3.5x faster than the 2nd binary.
In terms of instruction counts, the 1st binary requires a 4.3x smaller
count of instructions than the 2nd.
Q: Why is the 2nd instance slower?
A: Because of slower *initialization*.
Notice the two facts:
- the cause of the slowdown lies in the way Go deals with
initialization
- the slowdown is real
... AND IT IS THIS SLOWDOWN THAT I WANTED TO PREVENT FROM HAPPENING AS
THE EVOLUTION OF Go PROGRAMS PROCEEDS FROM SIMPLER ONES TO MORE
COMPLEX ONES.
So, my primary reasons for making my suggestions about compile-time
evaluation were related to devising a method that would prevent these
kinds of slowdowns.
But there you have it - the slowdown I mean. It is *real*, it is
actually happening. A couple of months ago I sort-of predicted that
this would happen. The Go language does not have a consistent
mechanism for dealing with these kinds of situations. So back then I
tried to suggest a mechanism that would make these issues
solvable. ... But I am *not* saying that what I suggested is the only
method for dealing with these issues. Of course, the method involving
Makefiles that others/you spoke/speak about is a valid solution as
well. That said, there does exist an argument against using the
Makefile-based method, the argument being: it did not happen. I mean,
look at the numbers I gave here. The slowdown is real. Was the
Makefile-based method used to remove this slowdown? No, it wasn't. The
Makefile-based method might look like a valid solution in theory, but
the historical record clearly shows that Go developers don't use it.
On May 8, 10:05 pm, "Paolo 'Blaisorblade' Giarrusso"
Jesper Louis Andersen
> I compiled this program with multiple versions of Go. The versions I
> will be talking about here are from the following dates: 2010.feb.09
> and 2010.apr.19
>
> I then run the binary like this:
do a 1000 runs say. The first couple are different and then it
stabilizes. My hypothesis is that CPU frequency scaling on the laptop
kicks in and makes the number funny. Making 1000 runs in quick
succession with each program and then using a kernel density plot of
the two suggest they are more or less the same. PNG for your
amusement:
https://docs.google.com/leaf?id=0B1kYTopXUcoJOGI0N2U0NTctOWI0NC00YTFiLTlkYWMtNzkwNTFlOTc2ZDk5&hl=en
--
J.
⚛
<jesper.louis.ander...@gmail.com> wrote:
> > I compiled this program with multiple versions of Go. The versions I
> > will be talking about here are from the following dates: 2010.feb.09
> > and 2010.apr.19
copying them one character at a time. The correct dates are:
hi-2010.02.19
hi-2010.04.28
If you look at the lines "perf stat ..." in my previous post, those
contain the correct dates.
Anyway, I checked the today's Go repository and the result it gives me
is this:
$ perf stat -- ./hi-2010.05.09
1.630863 task-clock-msecs # 0.882 CPUs
2693102 cycles # 1651.336 M/sec
2335643 instructions # 0.867 IPC
If I re-run (note: not recompile) the 1st example:
$ perf stat -- ./hi-2010.02.19
2797648 cycles # 1648.811 M/sec
2477286 instructions # 0.885 IPC
If you compare this (2477286 instructions) to my previous post
(2477098 instructions), you can see that the measurement is very
stable. If you are getting erratic numbers, I suppose the problem is
local to your computer only. (Maybe you should change the power saving
profile on your notebook, and obviously do not run any CPU-intensive
processes while doing the measurement.)
Yes, by looking at the "hi-2010.05.09" results and comparing them to
"hi-2010.04.28", it seems that the slowdown was only temporary. I
checked the revision "release.2010-04-27", and it gives me the results
I measured on "hi-2010.04.28". So, the question is *not* whether the
slowdown actually was there, but (1) why it occurred and (2) how it
later got fixed. By simply diff-ing the "release.2010-04-27" with
todays repository revision, I am unable to identify the cause by
myself. In other words, the cause seems to be there, somewhere, but it
is non-obvious. I don't know what exactly caused the slowdown, but I
am 100% positive that it is related to importing the "fmt" package.
Based on some other numbers from tests I see I made in the past, it
seems reasonable to deduce that it is related to package
initialization code. Otherwise than that, I am unsure what the actual
cause of the slowdown was.
> > I then run the binary like this:
>
> I can't reproduce here. Also, the numbers are quite erratic unless you
> do a 1000 runs say. The first couple are different and then it
> stabilizes. My hypothesis is that CPU frequency scaling on the laptop
> kicks in and makes the number funny. Making 1000 runs in quick
> succession with each program and then using a kernel density plot of
> the two suggest they are more or less the same. PNG for your
> amusement:
>
>
> --
> J.
Paolo 'Blaisorblade' Giarrusso
easily solved by something similar to V8 heap snapshotting, for
initialization having no external effects (i.e. no syscalls). And
that's vastly easier to implement.
Maybe this needs support at the language level, maybe this can be
avoided by recording the syscalls executed during initialization.
On 9 Mag, 15:18, ⚛ <0xe2.0x9a.0...@gmail.com> wrote:
> I don't know what exactly caused the slowdown, but I
> am 100% positive that it is related to importing the "fmt" package.
> Based on some other numbers from tests I see I made in the past, it
> seems reasonable to deduce that it is related to package
> initialization code.
utter nonsense. The relative majority of the executed instructions are
seen when adding Printf, then it comes the runtime init, and only
finally we have the fmt import.
I think that the instruction counts include instructions executed in
kernel, and the 43 executed syscalls easily account for a big part of
the initialization costs (and I don't think that cost can be removed).
> Otherwise than that, I am unsure what the actual
> cause of the slowdown was.
That's what happens in Java, for instance, so in that case supplying
the programmer compile-time evaluation would not save anything.
In any case, if the slowdown was temporary, it was caused by some
performance bug. Bisection search (through hg bisect) will find the
exact revision in logarithmic time.
Tons of evidence across the world show that program startup time is
influenced by:
1) binary load time (i.e. I/O, often non in sequential order)
2) binary relocation times for ELF libraries (missing here for now due
to static linking)
3) non profiled initialization routines
Also, tons of experience (rooting back to Donald Knuth and his saying
against premature optimization) show that without measurements you
can't be sure of anything. Even the most experienced developers are
often surprised by seeing how many great optimizations have unexpected
effects (including no effect at all, no matter how cool they were).
== EVIDENCE ==
Below benchmarks show that empty.go executes 611604 instructions,
import-fmt 1056630 instructions, hi executes 1804751 instructions.
6g -V tells:
6g version 5440
hg log shows that that revision is named "release.2010-05-04".
empty.go is:
package main
func main() {
}
hi.go is your original code.
import-fmt.go is quite complicated (it must use somehow the fmt
package, otherwise compilation is refused). Note it does not run any
code from fmt, it just takes the pointer to fmt.Printf.
package main
import (
"fmt"
)
}
func main() {
v := fmt.Printf
f(v)
}
$ perf stat --repeat 100 ../empty/empty
Performance counter stats for '../empty/empty' (100 runs):
0.522586 task-clock-msecs # 0.739 CPUs ( +-
3.517% )
0 context-switches # 0.000 M/sec ( +-
100.000% )
0 CPU-migrations # 0.000 M/sec
( +- nan% )
85 page-faults # 0.162 M/sec ( +-
0.084% )
768684 cycles # 1470.925 M/sec ( +-
0.400% )
611604 instructions # 0.796 IPC ( +-
0.078% )
131377 branches # 251.398 M/sec ( +-
0.074% )
2638 branch-misses # 2.008 % ( +-
0.607% )
<not counted> cache-
references
<not counted> cache-
misses
0.000707563 seconds time elapsed ( +- 3.539% )
$ perf stat --repeat 100 import-fmt
Performance counter stats for 'import-fmt' (100 runs):
0.711953 task-clock-msecs # 0.709 CPUs ( +-
2.603% )
0 context-switches # 0.000 M/sec
( +- nan% )
0 CPU-migrations # 0.000 M/sec
( +- nan% )
172 page-faults # 0.241 M/sec ( +-
0.039% )
1308506 cycles # 1837.910 M/sec ( +-
0.933% )
1056630 instructions # 0.808 IPC ( +-
0.052% )
217922 branches # 306.090 M/sec ( +-
0.055% )
4253 branch-misses # 1.952 % ( +-
0.401% )
<not counted> cache-
references
<not counted> cache-
misses
0.001003510 seconds time elapsed ( +- 2.710% )
$ perf stat --repeat 100 ../hi/hi
[...Program output...]
Performance counter stats for '../hi/hi' (100 runs):
1.282401 task-clock-msecs # 0.805 CPUs ( +-
3.080% )
0 context-switches # 0.000 M/sec
( +- nan% )
0 CPU-migrations # 0.000 M/sec
( +- nan% )
197 page-faults # 0.153 M/sec ( +-
0.040% )
2172984 cycles # 1694.465 M/sec ( +-
0.637% )
1804751 instructions # 0.831 IPC ( +-
0.038% )
409901 branches # 319.636 M/sec ( +-
0.031% )
14248 branch-misses # 3.476 % ( +-
0.167% )
<not counted> cache-references
<not counted> cache-misses
0.001592840 seconds time elapsed ( +- 9.130% )
⚛
> First, I'd like to point out that slow initialization could be more
> easily solved by something similar to V8 heap snapshotting, for
> initialization having no external effects (i.e. no syscalls). And
> that's vastly easier to implement.
> Maybe this needs support at the language level, maybe this can be
> avoided by recording the syscalls executed during initialization.
> On 9 Mag, 15:18, ⚛ <0xe2.0x9a.0...@gmail.com> wrote:> I don't know what exactly caused the slowdown, but I
> > am 100% positive that it is related to importing the "fmt" package.
> > Based on some other numbers from tests I see I made in the past, it
> > seems reasonable to deduce that it is related to package
> > initialization code.
>
> And based on the less-than-anecdotical evidence shown below, that's
> utter nonsense. The relative majority of the executed instructions are
> seen when adding Printf, then it comes the runtime init, and only
> finally we have the fmt import.
- runtime: 611604
- fmt import: 1056630-611604 = 445026
- initial call to Printf: 1804751-1056630 = 748121
That is your interpretation.
Here is a more objective interpretation of the number of instructions
that need to be *inevitably* executed:
- runtime: interval (0...611604)
- fmt import: interval (0...445026)
- initial call to Printf: interval (0...748121)
Having zero instructions as the lower bound is of course non-sense. So
let's use a variable there:
- runtime: interval (RT_MIN ... 611604)
- fmt import: interval (FMT_MIN ... 445026)
- initial call to Printf: interval (PRINTF_MIN ... 748121)
The upper bound of the amount of instructions potentially wasted in
initialization codes is:
WASTE_MAX =
(611604-RT_MIN) +
(445026-FMT_MIN) +
(748121-PRINTF_MIN)
Q: Have you provided any estimates for the values of RT_MIN, FMT_MIN,
PRINTF_MIN?
A: No, you have not.
... and by "providing estimates" I mean some real numbers. Pseudo-
estimates in the form of "I believe bla bla bla" or "bla bla easily
accounts for bla bla" are not estimates, but religious beliefs.
> I think that the instruction counts include instructions executed in
> kernel, and the 43 executed syscalls easily account for a big part of
> the initialization costs (and I don't think that cost can be removed).
$ valgrind --tool=callgrind ./hi-2010.05.09
==2944== Events : Ir
==2944== Collected : 2036232
==2944==
==2944== I refs: 2,036,232
$ perf stat -e instructions -- ./hi-2010.05.09
2192772 instructions # 0.000 IPC
$ perf stat -e instructions:k -- ./hi-2010.05.09
252113 instructions # 0.000 IPC
$ perf stat -e instructions:u -- ./hi-2010.05.09
1942932 instructions # 0.000 IPC
Conclusion: The majority of instructions (approx. 85%) comes from
executing *user* code, not kernel code.